Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal

Context: The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to stri...

Descripción completa

Detalles Bibliográficos
Autores: Guevara Vega, Cathy, Bernárdez Jiménez, Beatriz, Cruz Risco, Margarita, Durán Toro, Amador, Ruiz Cortés, Antonio, Solari, M.
Tipo de recurso: artículo
Estado:Versión aceptada para publicación
Fecha de publicación:2024
País:España
Institución:Universidad de Sevilla (US)
Repositorio:idUS. Depósito de Investigación de la Universidad de Sevilla
OAI Identifier:oai:idus.us.es:11441/163089
Acceso en línea:https://hdl.handle.net/11441/163089
https://doi.org/10.1016/j.jss.2024.112187
Access Level:acceso abierto
Palabra clave:Research artifact
Laboratory package
Replication package
Empirical software engineering
Human-oriented experiments
Open science
ACM(Association for Computing Machinery)
Badging policy
id ES_ce1cc5ab00a281d3db13c0f3a8b7f520
oai_identifier_str oai:idus.us.es:11441/163089
network_acronym_str ES
network_name_str España
repository_id_str
dc.title.none.fl_str_mv Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
title Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
spellingShingle Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
Guevara Vega, Cathy
Research artifact
Laboratory package
Replication package
Empirical software engineering
Human-oriented experiments
Open science
ACM(Association for Computing Machinery)
Badging policy
title_short Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
title_full Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
title_fullStr Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
title_full_unstemmed Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
title_sort Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
dc.creator.none.fl_str_mv Guevara Vega, Cathy
Bernárdez Jiménez, Beatriz
Cruz Risco, Margarita
Durán Toro, Amador
Ruiz Cortés, Antonio
Solari, M.
author Guevara Vega, Cathy
author_facet Guevara Vega, Cathy
Bernárdez Jiménez, Beatriz
Cruz Risco, Margarita
Durán Toro, Amador
Ruiz Cortés, Antonio
Solari, M.
author_role author
author2 Bernárdez Jiménez, Beatriz
Cruz Risco, Margarita
Durán Toro, Amador
Ruiz Cortés, Antonio
Solari, M.
author2_role author
author
author
author
author
dc.contributor.none.fl_str_mv Lenguajes y Sistemas Informáticos
dc.subject.none.fl_str_mv Research artifact
Laboratory package
Replication package
Empirical software engineering
Human-oriented experiments
Open science
ACM(Association for Computing Machinery)
Badging policy
topic Research artifact
Laboratory package
Replication package
Empirical software engineering
Human-oriented experiments
Open science
ACM(Association for Computing Machinery)
Badging policy
description Context: The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to strict protocols. Unfortunately, in Software Engineering (SE), this process is carried out in a non-systematic way, resulting in incomplete or inaccurate material shared by researchers, which hinders the reproducibility and replicability of empirical studies. Nevertheless, in recent years, it seems that the Empirical Software Engineering (ESE) community is embracing some of the proposed OS initiatives, such as the one proposed by the Association for Computing Machinery (ACM), which provides a badge system to evaluate the quality of a research artifact. This badge system has been adopted by several SE conferences as a method of assessing research artifacts. Aims: Focusing on human-oriented experiments (HOEs) in SE, whose research artifacts are more complex than those for computational experiments, this work applies Design Science Research (DSR) with a twofold purpose: (i) review the current status of HOEs research artifacts publication through evaluation of this practice in the most relevant ESE journals , and (ii) propose a structured outline for HOEs research artifacts driven by the aforementioned ACM badging policy. Method: Regarding the first purpose, we carried out a survey to analyze the current status of the publication of research artifacts considering relevant peer review journals and the quality of 106 research artifacts published in these journals with respect to the ACM badging policy. For the second purpose, an iterative process was carried out to review the content of 106 research artifacts research and their concordance with ACM badges, obtaining a structured scheme for HOEs research artifacts that has been validated through a detailed review of 12 research artifacts obtained from some of those of ACM badges in relevant SE conferences. In addition, we validated the proposal in the research artifacts of 2 of our own experiments. Results: Our survey reveals issues such as the 39,70% of journal studies making completely accessible their research artifacts; most of the analyzed research artifacts are incomplete; the most common repositories used in the ESE community to share the research artifacts are GitHub, institutional repositories, and Zenodo. On the other hand, the validated and structured research artifact outline consists of a list of ordered sections containing a set of artifacts, which can be mandatory or not to achieve a particular ACM badge. For its internal validation, several improvement iterations on the first release of the outline have been carried out based on the conference guidelines, the ACM badging policy, and other relevant proposals. Conclusions: Although the ESE community is making great efforts in standardization, review, and digital publishing related to OS, the availability and completeness of research artifacts can be improved. Our proposal for the elaboration of structured research artifact outline meets the requirements of HOEs in SE. Nevertheless, further research is needed not only to improve and externally validate it but also to disseminate its use among the research community.
publishDate 2024
dc.date.none.fl_str_mv 2024
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/acceptedVersion
format article
status_str acceptedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/11441/163089
https://doi.org/10.1016/j.jss.2024.112187
url https://hdl.handle.net/11441/163089
https://doi.org/10.1016/j.jss.2024.112187
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv JOURNAL OF SYSTEMS AND SOFTWARE, 218 (112187), 1-16.
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv ELSEVIER SCIENCE INC
publisher.none.fl_str_mv ELSEVIER SCIENCE INC
dc.source.none.fl_str_mv reponame:idUS. Depósito de Investigación de la Universidad de Sevilla
instname:Universidad de Sevilla (US)
instname_str Universidad de Sevilla (US)
reponame_str idUS. Depósito de Investigación de la Universidad de Sevilla
collection idUS. Depósito de Investigación de la Universidad de Sevilla
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869419939394224128
spelling Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposalGuevara Vega, CathyBernárdez Jiménez, BeatrizCruz Risco, MargaritaDurán Toro, AmadorRuiz Cortés, AntonioSolari, M.Research artifactLaboratory packageReplication packageEmpirical software engineeringHuman-oriented experimentsOpen scienceACM(Association for Computing Machinery)Badging policyContext: The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to strict protocols. Unfortunately, in Software Engineering (SE), this process is carried out in a non-systematic way, resulting in incomplete or inaccurate material shared by researchers, which hinders the reproducibility and replicability of empirical studies. Nevertheless, in recent years, it seems that the Empirical Software Engineering (ESE) community is embracing some of the proposed OS initiatives, such as the one proposed by the Association for Computing Machinery (ACM), which provides a badge system to evaluate the quality of a research artifact. This badge system has been adopted by several SE conferences as a method of assessing research artifacts. Aims: Focusing on human-oriented experiments (HOEs) in SE, whose research artifacts are more complex than those for computational experiments, this work applies Design Science Research (DSR) with a twofold purpose: (i) review the current status of HOEs research artifacts publication through evaluation of this practice in the most relevant ESE journals , and (ii) propose a structured outline for HOEs research artifacts driven by the aforementioned ACM badging policy. Method: Regarding the first purpose, we carried out a survey to analyze the current status of the publication of research artifacts considering relevant peer review journals and the quality of 106 research artifacts published in these journals with respect to the ACM badging policy. For the second purpose, an iterative process was carried out to review the content of 106 research artifacts research and their concordance with ACM badges, obtaining a structured scheme for HOEs research artifacts that has been validated through a detailed review of 12 research artifacts obtained from some of those of ACM badges in relevant SE conferences. In addition, we validated the proposal in the research artifacts of 2 of our own experiments. Results: Our survey reveals issues such as the 39,70% of journal studies making completely accessible their research artifacts; most of the analyzed research artifacts are incomplete; the most common repositories used in the ESE community to share the research artifacts are GitHub, institutional repositories, and Zenodo. On the other hand, the validated and structured research artifact outline consists of a list of ordered sections containing a set of artifacts, which can be mandatory or not to achieve a particular ACM badge. For its internal validation, several improvement iterations on the first release of the outline have been carried out based on the conference guidelines, the ACM badging policy, and other relevant proposals. Conclusions: Although the ESE community is making great efforts in standardization, review, and digital publishing related to OS, the availability and completeness of research artifacts can be improved. Our proposal for the elaboration of structured research artifact outline meets the requirements of HOEs in SE. Nevertheless, further research is needed not only to improve and externally validate it but also to disseminate its use among the research community.ELSEVIER SCIENCE INCLenguajes y Sistemas Informáticos2024info:eu-repo/semantics/articleinfo:eu-repo/semantics/acceptedVersionapplication/pdfapplication/pdfhttps://hdl.handle.net/11441/163089https://doi.org/10.1016/j.jss.2024.112187reponame:idUS. Depósito de Investigación de la Universidad de Sevillainstname:Universidad de Sevilla (US)InglésJOURNAL OF SYSTEMS AND SOFTWARE, 218 (112187), 1-16.info:eu-repo/semantics/openAccessoai:idus.us.es:11441/1630892026-06-17T12:51:07Z
score 15.811543

Similares en LA Referencia