Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation

Context: Metamorphic testing (MT) is a well-known approach to address the oracle problem in software testing. It does so by using expected relations – called metamorphic relations (MRs) – between the inputs and outputs of multiple system executions as test oracles. This approach involves identifying...

ver descrição completa

Detalhes bibliográficos
Autores: Cañizares, Pablo C., Gómez Abajo, Pablo, Guerra Sánchez, Esther, Lara Jaramillo, Juan de
Formato: artículo
Fecha de publicación:2026
País:España
Recursos:Universidad Autónoma de Madrid
Repositorio:Biblos-e Archivo. Repositorio Institucional de la UAM
Idioma:inglés
OAI Identifier:oai:dnet:biblosearchi::73d9f4152dcc36d171a61f3370102f7e
Acesso em linha:https://hdl.handle.net/10486/764341
https://dx.doi.org/10.1016/j.infsof.2026.108150
Access Level:acceso abierto
Palavra-chave:Metamorphic testing
Conversational assistants
Large language models
AI-based testing
Model-driven engineering
Informática
id ES_ae4832a107f2d643098a94641ed3e080
oai_identifier_str oai:dnet:biblosearchi::73d9f4152dcc36d171a61f3370102f7e
network_acronym_str ES
network_name_str España
repository_id_str
spelling Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generationCañizares, Pablo C.Gómez Abajo, PabloGuerra Sánchez, EstherLara Jaramillo, Juan deMetamorphic testingConversational assistantsLarge language modelsAI-based testingModel-driven engineeringInformáticaContext: Metamorphic testing (MT) is a well-known approach to address the oracle problem in software testing. It does so by using expected relations – called metamorphic relations (MRs) – between the inputs and outputs of multiple system executions as test oracles. This approach involves identifying meaningful MRs for the domain, defining them in an executable language, and creating suitable system inputs to check their satisfaction. However, these tasks are manual and demand substantial effort from testers. Objective: To reduce the effort of applying MT, we propose automating some tasks in the MT process with intelligent assistants, focussing on MR inference and generation of follow-up test cases. Methods: We propose a taxonomy of MT tasks amenable to LLM-based assistance, and an extensible architecture for their automation. The MT tasks are accessible via a conversational assistant, and may encompass validation and fixing cycles to improve the quality of the assistance. We have integrated the architecture with the MT framework Gotten. Results: We have evaluated our approach on the two considered MT tasks across three domains (data centres, autonomous vehicles, finite automata). We found the assistant effective at generating new MRs (between 2 and 6 depending on the domain) and follow-up test cases (with over 99.6% correctness). We also measured diversity and redundancy of the generated MRs, the benefits of fixing cycles on different LLMs when generating follow-up test cases, and performance degradation and computational cost as input complexity increases. Conclusions: Our architecture effectively automates two central MT tasks, namely, inferring MRs and generating follow-up test cases, showing potential to reduce the effort from MT practitioners and to automate other MT tasksWork funded by the Spanish MICINN with projects PID2021122270OB-I00, PID2024-155231OB-I00 and RED2022-134647-T, and by the Community of Madrid’s PRICIT programme with project 2025/00097/001ElsevierDepartamento de Ingeniería InformáticaEscuela Politécnica SuperiorGobierno de España20262026-04-13research articlehttp://purl.org/coar/resource_type/c_2df8fbb1VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/10486/764341https://dx.doi.org/10.1016/j.infsof.2026.108150reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2Attribution-NonCommercial-NoDerivatives 4.0 Internationalhttp://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:dnet:biblosearchi::73d9f4152dcc36d171a61f3370102f7e2026-06-23T12:46:27Z
dc.title.none.fl_str_mv Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
title Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
spellingShingle Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
Cañizares, Pablo C.
Metamorphic testing
Conversational assistants
Large language models
AI-based testing
Model-driven engineering
Informática
title_short Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
title_full Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
title_fullStr Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
title_full_unstemmed Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
title_sort Towards metamorphic testing with LLM-based workflows: Metamorphic relation inference and follow-up test case generation
dc.creator.none.fl_str_mv Cañizares, Pablo C.
Gómez Abajo, Pablo
Guerra Sánchez, Esther
Lara Jaramillo, Juan de
author Cañizares, Pablo C.
author_facet Cañizares, Pablo C.
Gómez Abajo, Pablo
Guerra Sánchez, Esther
Lara Jaramillo, Juan de
author_role author
author2 Gómez Abajo, Pablo
Guerra Sánchez, Esther
Lara Jaramillo, Juan de
author2_role author
author
author
dc.contributor.none.fl_str_mv Departamento de Ingeniería Informática
Escuela Politécnica Superior
Gobierno de España
dc.subject.none.fl_str_mv Metamorphic testing
Conversational assistants
Large language models
AI-based testing
Model-driven engineering
Informática
topic Metamorphic testing
Conversational assistants
Large language models
AI-based testing
Model-driven engineering
Informática
description Context: Metamorphic testing (MT) is a well-known approach to address the oracle problem in software testing. It does so by using expected relations – called metamorphic relations (MRs) – between the inputs and outputs of multiple system executions as test oracles. This approach involves identifying meaningful MRs for the domain, defining them in an executable language, and creating suitable system inputs to check their satisfaction. However, these tasks are manual and demand substantial effort from testers. Objective: To reduce the effort of applying MT, we propose automating some tasks in the MT process with intelligent assistants, focussing on MR inference and generation of follow-up test cases. Methods: We propose a taxonomy of MT tasks amenable to LLM-based assistance, and an extensible architecture for their automation. The MT tasks are accessible via a conversational assistant, and may encompass validation and fixing cycles to improve the quality of the assistance. We have integrated the architecture with the MT framework Gotten. Results: We have evaluated our approach on the two considered MT tasks across three domains (data centres, autonomous vehicles, finite automata). We found the assistant effective at generating new MRs (between 2 and 6 depending on the domain) and follow-up test cases (with over 99.6% correctness). We also measured diversity and redundancy of the generated MRs, the benefits of fixing cycles on different LLMs when generating follow-up test cases, and performance degradation and computational cost as input complexity increases. Conclusions: Our architecture effectively automates two central MT tasks, namely, inferring MRs and generating follow-up test cases, showing potential to reduce the effort from MT practitioners and to automate other MT tasks
publishDate 2026
dc.date.none.fl_str_mv 2026
2026-04-13
dc.type.none.fl_str_mv research article
http://purl.org/coar/resource_type/c_2df8fbb1
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/10486/764341
https://dx.doi.org/10.1016/j.infsof.2026.108150
url https://hdl.handle.net/10486/764341
https://dx.doi.org/10.1016/j.infsof.2026.108150
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivatives 4.0 International
http://creativecommons.org/licenses/by-nc-nd/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivatives 4.0 International
http://creativecommons.org/licenses/by-nc-nd/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:Biblos-e Archivo. Repositorio Institucional de la UAM
instname:Universidad Autónoma de Madrid
instname_str Universidad Autónoma de Madrid
reponame_str Biblos-e Archivo. Repositorio Institucional de la UAM
collection Biblos-e Archivo. Repositorio Institucional de la UAM
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869416548525932545
score 15,811543

Registos relacionados