A model-based approach for specifying changes in replications of empirical studies in computer Science

The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also repl...

Descripción completa

Detalles Bibliográficos
Autores: Cruz Risco, Margarita, Bernárdez Jiménez, Beatriz, Durán Toro, Amador, Guevara Vega, Cathy, Ruiz Cortés, Antonio
Tipo de recurso: artículo
Estado:Versión aceptada para publicación
Fecha de publicación:2023
País:España
Institución:Universidad de Sevilla (US)
Repositorio:idUS. Depósito de Investigación de la Universidad de Sevilla
OAI Identifier:oai:idus.us.es:11441/163076
Acceso en línea:https://hdl.handle.net/11441/163076
https://doi.org/10.1007/s00607-022-01133-x
Access Level:acceso abierto
Palabra clave:Replication changes
Templates
Linguistic patterns
Families of experiments
Threats to validity
id ES_eab7cddcb7fe62dd5b232e9b7348d6f5
oai_identifier_str oai:idus.us.es:11441/163076
network_acronym_str ES
network_name_str España
repository_id_str
spelling A model-based approach for specifying changes in replications of empirical studies in computer ScienceCruz Risco, MargaritaBernárdez Jiménez, BeatrizDurán Toro, AmadorGuevara Vega, CathyRuiz Cortés, AntonioReplication changesTemplatesLinguistic patternsFamilies of experimentsThreats to validityThe need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying Design Science Research, we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replication changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community.SPRINGER; SPRINGER WIENLenguajes y Sistemas Informáticos2023info:eu-repo/semantics/articleinfo:eu-repo/semantics/acceptedVersionapplication/pdfapplication/pdfhttps://hdl.handle.net/11441/163076https://doi.org/10.1007/s00607-022-01133-xreponame:idUS. Depósito de Investigación de la Universidad de Sevillainstname:Universidad de Sevilla (US)InglésComputing: archives for scientific computing, 105 (6), 1189-1213.info:eu-repo/semantics/openAccessoai:idus.us.es:11441/1630762026-06-17T12:51:07Z
dc.title.none.fl_str_mv A model-based approach for specifying changes in replications of empirical studies in computer Science
title A model-based approach for specifying changes in replications of empirical studies in computer Science
spellingShingle A model-based approach for specifying changes in replications of empirical studies in computer Science
Cruz Risco, Margarita
Replication changes
Templates
Linguistic patterns
Families of experiments
Threats to validity
title_short A model-based approach for specifying changes in replications of empirical studies in computer Science
title_full A model-based approach for specifying changes in replications of empirical studies in computer Science
title_fullStr A model-based approach for specifying changes in replications of empirical studies in computer Science
title_full_unstemmed A model-based approach for specifying changes in replications of empirical studies in computer Science
title_sort A model-based approach for specifying changes in replications of empirical studies in computer Science
dc.creator.none.fl_str_mv Cruz Risco, Margarita
Bernárdez Jiménez, Beatriz
Durán Toro, Amador
Guevara Vega, Cathy
Ruiz Cortés, Antonio
author Cruz Risco, Margarita
author_facet Cruz Risco, Margarita
Bernárdez Jiménez, Beatriz
Durán Toro, Amador
Guevara Vega, Cathy
Ruiz Cortés, Antonio
author_role author
author2 Bernárdez Jiménez, Beatriz
Durán Toro, Amador
Guevara Vega, Cathy
Ruiz Cortés, Antonio
author2_role author
author
author
author
dc.contributor.none.fl_str_mv Lenguajes y Sistemas Informáticos
dc.subject.none.fl_str_mv Replication changes
Templates
Linguistic patterns
Families of experiments
Threats to validity
topic Replication changes
Templates
Linguistic patterns
Families of experiments
Threats to validity
description The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying Design Science Research, we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replication changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community.
publishDate 2023
dc.date.none.fl_str_mv 2023
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/acceptedVersion
format article
status_str acceptedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/11441/163076
https://doi.org/10.1007/s00607-022-01133-x
url https://hdl.handle.net/11441/163076
https://doi.org/10.1007/s00607-022-01133-x
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Computing: archives for scientific computing, 105 (6), 1189-1213.
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv SPRINGER; SPRINGER WIEN
publisher.none.fl_str_mv SPRINGER; SPRINGER WIEN
dc.source.none.fl_str_mv reponame:idUS. Depósito de Investigación de la Universidad de Sevilla
instname:Universidad de Sevilla (US)
instname_str Universidad de Sevilla (US)
reponame_str idUS. Depósito de Investigación de la Universidad de Sevilla
collection idUS. Depósito de Investigación de la Universidad de Sevilla
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869423164175417344
score 15,811543

Similares en LA Referencia