Evaluating Gender Bias on Machine Translation using Large Language Models
This thesis evaluates gender bias in Machine Translation using Large Language Models. A comprehensive benchmarking process assessed various encoder-decoder NMT models and decoder-only base LLMs for English-to-Catalan and English-to-Spanish translations. Distinct test sets, including FLoRes, WinoMT,...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2024 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/410885 |
| Acceso en línea: | https://hdl.handle.net/2117/410885 |
| Access Level: | acceso abierto |
| Palabra clave: | Machine translating Large Language Models Machine Translation Gender Bias Prompting Engineering Traducció automàtica Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústic |
| Sumario: | This thesis evaluates gender bias in Machine Translation using Large Language Models. A comprehensive benchmarking process assessed various encoder-decoder NMT models and decoder-only base LLMs for English-to-Catalan and English-to-Spanish translations. Distinct test sets, including FLoRes, WinoMT, Gold BUG, and MuST-SHE, were employed to evaluate translation quality and gender bias, obtaining metrics such as BLEU, COMET, Gender Accuracy, F1-male, F1-female, ?G, and ?S. Results indicated the presence of gender bias across all models, with base LLMs exhibiting more bias than NMT models. To mitigate this bias, prompting engineering techniques were implemented in an instruction-tuned LLM. After multiple attempts, a prompt resulting in a significant reduction in gender bias across the test sets was identified, achieving remarkable gender scores. The prompt followed a simplified step-by-step approach with 5-shots on anti-stereotypical content and increased female representation. This prompt enabled the instructed LLM to perform competitively, achieving results comparable to NMT models, even surpassing some of them. However, a loss in translation quality using this prompt was observed. |
|---|