Meta-Fair: AI-assisted fairness testing of large language models

Context: Fairness — the absence of unjustified bias — is a core principle in the development of Artificial Intelligence (AI) systems, yet it remains difficult to assess and enforce. Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed template...

Descripción completa

Detalles Bibliográficos
Autores: Romero Arjona, Miguel, Parejo Maestre, José Antonio, Alonso Valenzuela, Juan Carlos, Sánchez Jerez, Ana Belén, Arrieta, Aitor, Segura Rueda, Sergio
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2026
País:España
Institución:Universidad de Sevilla (US)
Repositorio:idUS. Depósito de Investigación de la Universidad de Sevilla
OAI Identifier:oai:dnet:idus________::395c050828d2324184a5206a19baa719
Acceso en línea:https://hdl.handle.net/11441/186356
https://doi.org/10.1016/j.infsof.2026.108075
Access Level:acceso abierto
Palabra clave:Metamorphic testing
Large language models
Fairness
Bias
Artificial intelligence
id ES_4dfa896acbf95bc87b68e23f70d2f238
oai_identifier_str oai:dnet:idus________::395c050828d2324184a5206a19baa719
network_acronym_str ES
network_name_str España
repository_id_str
spelling Meta-Fair: AI-assisted fairness testing of large language modelsRomero Arjona, MiguelParejo Maestre, José AntonioAlonso Valenzuela, Juan CarlosSánchez Jerez, Ana BelénArrieta, AitorSegura Rueda, SergioMetamorphic testingLarge language modelsFairnessBiasArtificial intelligenceContext: Fairness — the absence of unjustified bias — is a core principle in the development of Artificial Intelligence (AI) systems, yet it remains difficult to assess and enforce. Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed templates, deterministic heuristics, and curated datasets, making them resource-intensive and difficult to scale. Objectives: This work aims to lay the groundwork for a novel, automated method for testing fairness in LLMs, reducing the dependence on domain-specific resources and broadening the applicability of current approaches. Methods: Our approach, Meta-Fair, is based on two key ideas. First, we adopt metamorphic testing to uncover bias by examining how model outputs vary in response to controlled modifications of input prompts, defined by metamorphic relations (MRs). Second, we propose exploiting the potential of LLMs for both test case generation and output evaluation, leveraging their capability to generate diverse inputs and classify outputs effectively. The proposal is complemented by three open-source tools supporting LLM-driven generation, execution, and evaluation of test cases. Results: We report the findings of several experiments involving 12 pre-trained LLMs, 14 MRs, 5 bias dimensions, and 7.9K automatically generated test cases. The results show that Meta-Fair is effective in uncovering bias in LLMs, achieving an average precision of 92% and revealing biased behaviour in 29% of executions. Additionally, LLMs prove to be reliable and consistent evaluators, with the best-performing models achieving F1-scores of up to 0.79. Although non-determinism affects consistency, these effects can be mitigated through careful MR design. Conclusion: This work highlights the feasibility and potential of integrating metamorphic testing with LLMdriven test generation and assessment. While challenges remain to ensure broader applicability, the results indicate a promising path towards an unprecedented level of automation in LLM testingElsevierLenguajes y Sistemas InformáticosTIC205: Ingeniería del Software AplicadaMinisterio de Ciencia e Innovación (MICIN). EspañaMinisterio de Ciencia, Innovación y Universidades (MICIU). España2026info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttps://hdl.handle.net/11441/186356https://doi.org/10.1016/j.infsof.2026.108075reponame:idUS. Depósito de Investigación de la Universidad de Sevillainstname:Universidad de Sevilla (US)InglésInformation and Software Technology, 194. PID2021-126227NB-C22TED2021-131023B-C21PID2024-156482NB-I00https://www.sciencedirect.com/science/article/pii/S0950584926000649?via%3Dihubinfo:eu-repo/semantics/openAccessoai:dnet:idus________::395c050828d2324184a5206a19baa7192026-06-17T12:51:07Z
dc.title.none.fl_str_mv Meta-Fair: AI-assisted fairness testing of large language models
title Meta-Fair: AI-assisted fairness testing of large language models
spellingShingle Meta-Fair: AI-assisted fairness testing of large language models
Romero Arjona, Miguel
Metamorphic testing
Large language models
Fairness
Bias
Artificial intelligence
title_short Meta-Fair: AI-assisted fairness testing of large language models
title_full Meta-Fair: AI-assisted fairness testing of large language models
title_fullStr Meta-Fair: AI-assisted fairness testing of large language models
title_full_unstemmed Meta-Fair: AI-assisted fairness testing of large language models
title_sort Meta-Fair: AI-assisted fairness testing of large language models
dc.creator.none.fl_str_mv Romero Arjona, Miguel
Parejo Maestre, José Antonio
Alonso Valenzuela, Juan Carlos
Sánchez Jerez, Ana Belén
Arrieta, Aitor
Segura Rueda, Sergio
author Romero Arjona, Miguel
author_facet Romero Arjona, Miguel
Parejo Maestre, José Antonio
Alonso Valenzuela, Juan Carlos
Sánchez Jerez, Ana Belén
Arrieta, Aitor
Segura Rueda, Sergio
author_role author
author2 Parejo Maestre, José Antonio
Alonso Valenzuela, Juan Carlos
Sánchez Jerez, Ana Belén
Arrieta, Aitor
Segura Rueda, Sergio
author2_role author
author
author
author
author
dc.contributor.none.fl_str_mv Lenguajes y Sistemas Informáticos
TIC205: Ingeniería del Software Aplicada
Ministerio de Ciencia e Innovación (MICIN). España
Ministerio de Ciencia, Innovación y Universidades (MICIU). España
dc.subject.none.fl_str_mv Metamorphic testing
Large language models
Fairness
Bias
Artificial intelligence
topic Metamorphic testing
Large language models
Fairness
Bias
Artificial intelligence
description Context: Fairness — the absence of unjustified bias — is a core principle in the development of Artificial Intelligence (AI) systems, yet it remains difficult to assess and enforce. Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed templates, deterministic heuristics, and curated datasets, making them resource-intensive and difficult to scale. Objectives: This work aims to lay the groundwork for a novel, automated method for testing fairness in LLMs, reducing the dependence on domain-specific resources and broadening the applicability of current approaches. Methods: Our approach, Meta-Fair, is based on two key ideas. First, we adopt metamorphic testing to uncover bias by examining how model outputs vary in response to controlled modifications of input prompts, defined by metamorphic relations (MRs). Second, we propose exploiting the potential of LLMs for both test case generation and output evaluation, leveraging their capability to generate diverse inputs and classify outputs effectively. The proposal is complemented by three open-source tools supporting LLM-driven generation, execution, and evaluation of test cases. Results: We report the findings of several experiments involving 12 pre-trained LLMs, 14 MRs, 5 bias dimensions, and 7.9K automatically generated test cases. The results show that Meta-Fair is effective in uncovering bias in LLMs, achieving an average precision of 92% and revealing biased behaviour in 29% of executions. Additionally, LLMs prove to be reliable and consistent evaluators, with the best-performing models achieving F1-scores of up to 0.79. Although non-determinism affects consistency, these effects can be mitigated through careful MR design. Conclusion: This work highlights the feasibility and potential of integrating metamorphic testing with LLMdriven test generation and assessment. While challenges remain to ensure broader applicability, the results indicate a promising path towards an unprecedented level of automation in LLM testing
publishDate 2026
dc.date.none.fl_str_mv 2026
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/11441/186356
https://doi.org/10.1016/j.infsof.2026.108075
url https://hdl.handle.net/11441/186356
https://doi.org/10.1016/j.infsof.2026.108075
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Information and Software Technology, 194.
PID2021-126227NB-C22
TED2021-131023B-C21
PID2024-156482NB-I00
https://www.sciencedirect.com/science/article/pii/S0950584926000649?via%3Dihub
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:idUS. Depósito de Investigación de la Universidad de Sevilla
instname:Universidad de Sevilla (US)
instname_str Universidad de Sevilla (US)
reponame_str idUS. Depósito de Investigación de la Universidad de Sevilla
collection idUS. Depósito de Investigación de la Universidad de Sevilla
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869407727170617344
score 15,81155

Similares en LA Referencia