Author & Reviewer

NLP-Telescope: Extending MT-Telescope for Natural Language Processing Models 

Details: The analysis of a language model requires an useful evaluation tool that offers an user-friendly interface and suitable analysis functionalities. MT-Telescope is one of these tools that analyses and compares the results of two machine translation systems given one reference and that performs a detailed analysis of these systems. The tool offers a variety of evaluation metrics and the study of linguistic phenomena. However, it would be interesting to allow the evaluation of other Natural Language Processing tasks such as Text Summarization, Dialogue System and General Classification Tasks, in addition to Machine Translation. It would also be of interest for the tool to be able to evaluate N systems from M references (in which N and M are higher than 1), instead of performing the evaluation of two systems based on a single reference, with the aim to perform the evaluation of biases in the systems (mainly gender bias in Machine Translation systems) and to perform the ranking of the systems based on some kind of an universal metric, aggregating all the selected metrics. This work consists on the creation of an evaluation tool that updates and expands MT-Telescope, designated as NLP-Telescope. NLP-Telescope is set to solve the previously described problems. It is concluded that NLP-Telescope is an indispensable tool to assist users on the evaluation of Natural Language Processing systems by allowing them to select one of three bias evaluation methods and ranking the systems based on an universal metric.

Keywords: Natural Language Processing, Evaluation Tool, Evaluation Metrics, Bias, Aggregation Mechanisms, Linguistic Phenomena