Machine translation evaluation metrics benchmarking: from traditional MT to LLMs

Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2022-2023. Tutor: Jordi Vitrià i Marca

Detalles Bibliográficos
Autor: López Caro, Álvaro
Tipo de recurso: tesis de maestría
Fecha de publicación:2023
País:España
Institución:Universidad de Barcelona
Repositorio:Dipòsit Digital de la UB
OAI Identifier:oai:diposit.ub.edu:2445/214303
Acceso en línea:https://hdl.handle.net/2445/214303
Access Level:acceso abierto
Palabra clave:Traducció automàtica
Lingüística computacional
Tractament del llenguatge natural (Informàtica)
Treballs de fi de màster
Machine translating
Computational linguistics
Natural language processing (Computer science)
Master's thesis
id ES_b406334fa80fdf3c84baaac3104f857c
oai_identifier_str oai:diposit.ub.edu:2445/214303
network_acronym_str ES
network_name_str España
repository_id_str
spelling Machine translation evaluation metrics benchmarking: from traditional MT to LLMsLópez Caro, ÁlvaroTraducció automàticaLingüística computacionalTractament del llenguatge natural (Informàtica)Treballs de fi de màsterMachine translatingComputational linguisticsNatural language processing (Computer science)Master's thesisTreballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2022-2023. Tutor: Jordi Vitrià i MarcaThis thesis endeavors to cast a spotlight on the evolution and applicability of machine translation (MT) evaluation metrics and models, mainly contrasting statistical methods against the more contemporary neural-based ones, where we also give special attention to the exciting modern Large Language Models (LLMs). MT, a significant area in Natural Language Processing (NLP), has seen a vast metamorphosis over the years, bringing into focus the critical need for thorough exploration of these evolving systems. Our research is anchored on the Digital Corpus of the European Parliament (DCEP), a complex and multilingual corpus that makes it an ideal testbed to benchmark MT models given its comprehensive and diversified linguistic data. Through the use of this extensive corpus, we aim to present a comprehensive benchmarking of various selected MT models, encapsulating not just their evolution but also their performance dynamics across different tasks and contexts. A vital facet of our study includes evaluating the relevance and reliability of various MT metrics, such as the old BLEU, METEOR, CHRF, along with newer neuralbased metrics which promise to capture semantics more effectively. We aim to uncover the inherent strengths and limitations of these metrics, consequently guiding the choice of appropriate metrics for specific MT contexts for future practitioners and researchers. In this holistic examination, we will also propose to analyze the interplay between model selection, evaluation metric, and translation quality. This thesis will provide a novel lens to understand the idiosyncrasies of various popular MT models and evaluation metrics, ultimately contributing to more effective and nuanced applications of MT. In sum, this exploration promises to furnish a new perspective on MT evaluation, honing our understanding of both the models’ and metrics’ evolutionary paths, and providing insights into their contextual performance on the DCEP corpus, creating a benchmark that can serve the broader MT community. The insights derived aim to significantly contribute to the latter. The reader can find all the code, used for the text pre/postprocessing and evaluation of the models and metrics at play along with other intermediate matters, published publicly in our GitHub repository.Vitrià i Marca, Jordi2023info:eu-repo/semantics/masterThesisapplication/pdfhttps://hdl.handle.net/2445/214303Màster Oficial - Fonaments de la Ciència de Dadesreponame:Dipòsit Digital de la UBinstname:Universidad de BarcelonaIngléscc-by-nc-nd (c) Álvaro López Caro, 2023codi: Apache (c) Álvaro López Caro, 2023http://creativecommons.org/licenses/by-nc-nd/3.0/es/https://www.apache.org/licenses/LICENSE-2.0.txtinfo:eu-repo/semantics/openAccessoai:diposit.ub.edu:2445/2143032026-05-27T06:46:51Z
dc.title.none.fl_str_mv Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
title Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
spellingShingle Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
López Caro, Álvaro
Traducció automàtica
Lingüística computacional
Tractament del llenguatge natural (Informàtica)
Treballs de fi de màster
Machine translating
Computational linguistics
Natural language processing (Computer science)
Master's thesis
title_short Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
title_full Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
title_fullStr Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
title_full_unstemmed Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
title_sort Machine translation evaluation metrics benchmarking: from traditional MT to LLMs
dc.creator.none.fl_str_mv López Caro, Álvaro
author López Caro, Álvaro
author_facet López Caro, Álvaro
author_role author
dc.contributor.none.fl_str_mv Vitrià i Marca, Jordi
dc.subject.none.fl_str_mv Traducció automàtica
Lingüística computacional
Tractament del llenguatge natural (Informàtica)
Treballs de fi de màster
Machine translating
Computational linguistics
Natural language processing (Computer science)
Master's thesis
topic Traducció automàtica
Lingüística computacional
Tractament del llenguatge natural (Informàtica)
Treballs de fi de màster
Machine translating
Computational linguistics
Natural language processing (Computer science)
Master's thesis
description Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2022-2023. Tutor: Jordi Vitrià i Marca
publishDate 2023
dc.date.none.fl_str_mv 2023
dc.type.none.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
dc.identifier.none.fl_str_mv https://hdl.handle.net/2445/214303
url https://hdl.handle.net/2445/214303
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.rights.none.fl_str_mv cc-by-nc-nd (c) Álvaro López Caro, 2023
codi: Apache (c) Álvaro López Caro, 2023
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
https://www.apache.org/licenses/LICENSE-2.0.txt
info:eu-repo/semantics/openAccess
rights_invalid_str_mv cc-by-nc-nd (c) Álvaro López Caro, 2023
codi: Apache (c) Álvaro López Caro, 2023
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
https://www.apache.org/licenses/LICENSE-2.0.txt
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv Màster Oficial - Fonaments de la Ciència de Dades
reponame:Dipòsit Digital de la UB
instname:Universidad de Barcelona
instname_str Universidad de Barcelona
reponame_str Dipòsit Digital de la UB
collection Dipòsit Digital de la UB
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869417226435559424
score 15,811543

Similares en LA Referencia