Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories

The research presented in this paper has been supported in part by the Government of Spain , through project “Dependentium” (PID2022-139551NB-I00).

Detalles Bibliográficos
Autores: Gonzalez-Barahona, Jesus M., Robles, Gregorio
Tipo de recurso: artículo
Fecha de publicación:2023
País:España
Institución:Universidad Rey Juan Carlos
Repositorio:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
OAI Identifier:oai:burjcdigital.urjc.es:10115/24912
Acceso en línea:https://hdl.handle.net/10115/24912
Access Level:acceso abierto
Palabra clave:Reproducible research
Mining software repositories
Reproducibility
Validation studies
Empirical software engineering
id ES_cdff99f466e4d6221fc377e3069ea890
oai_identifier_str oai:burjcdigital.urjc.es:10115/24912
network_acronym_str ES
network_name_str España
repository_id_str
spelling Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositoriesGonzalez-Barahona, Jesus M.Robles, GregorioReproducible researchMining software repositoriesReproducibilityValidation studiesEmpirical software engineeringThe research presented in this paper has been supported in part by the Government of Spain , through project “Dependentium” (PID2022-139551NB-I00).Context: In 2012, our paper ‘‘On the reproducibility of empirical software engineering studies based on data retrieved from development repositories’’ was published. It proposed a method for assessing the reproducibility of studies based on mining software repositories (MSR studies). Since then, several approaches have happened with respect to the study of the reproducibility of this kind of studies. Objective: To revisit the proposals of that paper, analyzing to which extent they remain valid, and how they relate to current initiatives and studies on reproducibility and validation of research results in empirical software engineering. Method: We analyze the most relevant studies affecting assumptions or consequences of the approach of the original paper, and other initiatives related to the evaluation of replicability aspects of empirical software engineering studies. We compare the results of that analysis with the results of the original study, finding similarities and differences. We also run a reproducibility assessment study on current MSR papers. Based on the comparison, and the applicability of the method to current papers, we draw conclusions on the validity of the approach of the original paper. Main lessons learned: The method proposed in the original paper is still valid, and compares well with other more recent methods. It matches the results of relevant studies on reproducibility, and a systematic comparison with them shows that our approach is aligned with their proposals. Our method has practical use, and complements well the current major initiatives on the review of reproducibility artifacts. As a side result, we learn that the reproducibility of MSR studies has improved during the last decade. Vision: We propose to use our approach as a fundamental element of a more profound review of the reproducibility of MSR studies, and of the characterization of validation studies in this realm.Elsevier202320232023info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/10115/24912reponame:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlosinstname:Universidad Rey Juan CarlosInglésAtribución 4.0 Internacionalhttp://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:burjcdigital.urjc.es:10115/249122026-06-24T12:48:17Z
dc.title.none.fl_str_mv Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
title Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
spellingShingle Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
Gonzalez-Barahona, Jesus M.
Reproducible research
Mining software repositories
Reproducibility
Validation studies
Empirical software engineering
title_short Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
title_full Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
title_fullStr Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
title_full_unstemmed Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
title_sort Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
dc.creator.none.fl_str_mv Gonzalez-Barahona, Jesus M.
Robles, Gregorio
author Gonzalez-Barahona, Jesus M.
author_facet Gonzalez-Barahona, Jesus M.
Robles, Gregorio
author_role author
author2 Robles, Gregorio
author2_role author
dc.subject.none.fl_str_mv Reproducible research
Mining software repositories
Reproducibility
Validation studies
Empirical software engineering
topic Reproducible research
Mining software repositories
Reproducibility
Validation studies
Empirical software engineering
description The research presented in this paper has been supported in part by the Government of Spain , through project “Dependentium” (PID2022-139551NB-I00).
publishDate 2023
dc.date.none.fl_str_mv 2023
2023
2023
dc.type.none.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/10115/24912
url https://hdl.handle.net/10115/24912
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.rights.none.fl_str_mv Atribución 4.0 Internacional
http://creativecommons.org/licenses/by/4.0/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv Atribución 4.0 Internacional
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
instname:Universidad Rey Juan Carlos
instname_str Universidad Rey Juan Carlos
reponame_str BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
collection BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869419928557191168
score 15,812429

Similares en LA Referencia