Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site

Brain MRI researchers conducting multisite studies, such as within the ENIGMA Consortium, are very aware of the importance of controlling the effects of the site (EoS) in the statistical analysis. Conversely, authors of the novel machine-learning MRI studies may remove the EoS when training the mach...

Descripción completa

Detalles Bibliográficos
Autores: Solanes, Aleix, Palau, Pol, Fortea, Lydia, Salvador, Raymond, González Navarro, Laura, Llach, Cristian, Valentí Ribas, Marc, Vieta i Pascual, Eduard, 1963-, Radua, Joaquim
Tipo de recurso: artículo
Estado:Versión aceptada para publicación
Fecha de publicación:2021
País:España
Institución:Universidad de Barcelona
Repositorio:Dipòsit Digital de la UB
OAI Identifier:oai:diposit.ub.edu:2445/219882
Acceso en línea:https://hdl.handle.net/2445/219882
Access Level:acceso abierto
Palabra clave:Aprenentatge automàtic
Estadística mèdica
Imatges per ressonància magnètica
Machine learning
Medical statistics
Magnetic resonance imaging
id ES_6e2ab3cbcb045bb49d9d7dfedfac4af4
oai_identifier_str oai:diposit.ub.edu:2445/219882
network_acronym_str ES
network_name_str España
repository_id_str
spelling Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the siteSolanes, AleixPalau, PolFortea, LydiaSalvador, RaymondGonzález Navarro, LauraLlach, CristianValentí Ribas, MarcVieta i Pascual, Eduard, 1963-Radua, JoaquimAprenentatge automàticEstadística mèdicaImatges per ressonància magnèticaMachine learningMedical statisticsMagnetic resonance imagingBrain MRI researchers conducting multisite studies, such as within the ENIGMA Consortium, are very aware of the importance of controlling the effects of the site (EoS) in the statistical analysis. Conversely, authors of the novel machine-learning MRI studies may remove the EoS when training the machine-learning models but not control them when estimating the models' accuracy, potentially leading to severely biased estimates. We show examples from a toy simulation study and real MRI data in which we remove the EoS from both the "training set" and the "test set" during the training and application of the model. However, the accuracy is still inflated (or occasionally shrunk) unless we further control the EoS during the estimation of the accuracy. We also provide several methods for controlling the EoS during the estimation of the accuracy, and a simple R package ("multisite.accuracy") that smoothly does this task for several accuracy estimates (e.g.,sensitivity/specificity, area under the curve, correlation, hazard ratio, etc.).Elsevier B.V.2021info:eu-repo/semantics/articleinfo:eu-repo/semantics/acceptedVersionapplication/pdfhttps://hdl.handle.net/2445/219882Articles publicats en revistes (Medicina)reponame:Dipòsit Digital de la UBinstname:Universidad de BarcelonaInglésVersió postprint del document publicat a: https://doi.org/10.1016/j.pscychresns.2021.111313Psychiatry Research-Neuroimaging, 2021, vol. 314https://doi.org/10.1016/j.pscychresns.2021.111313cc-by-nc-nd (c) Elsevier B.V., 2021http://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:diposit.ub.edu:2445/2198822026-05-27T06:46:51Z
dc.title.none.fl_str_mv Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
title Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
spellingShingle Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
Solanes, Aleix
Aprenentatge automàtic
Estadística mèdica
Imatges per ressonància magnètica
Machine learning
Medical statistics
Magnetic resonance imaging
title_short Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
title_full Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
title_fullStr Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
title_full_unstemmed Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
title_sort Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site
dc.creator.none.fl_str_mv Solanes, Aleix
Palau, Pol
Fortea, Lydia
Salvador, Raymond
González Navarro, Laura
Llach, Cristian
Valentí Ribas, Marc
Vieta i Pascual, Eduard, 1963-
Radua, Joaquim
author Solanes, Aleix
author_facet Solanes, Aleix
Palau, Pol
Fortea, Lydia
Salvador, Raymond
González Navarro, Laura
Llach, Cristian
Valentí Ribas, Marc
Vieta i Pascual, Eduard, 1963-
Radua, Joaquim
author_role author
author2 Palau, Pol
Fortea, Lydia
Salvador, Raymond
González Navarro, Laura
Llach, Cristian
Valentí Ribas, Marc
Vieta i Pascual, Eduard, 1963-
Radua, Joaquim
author2_role author
author
author
author
author
author
author
author
dc.subject.none.fl_str_mv Aprenentatge automàtic
Estadística mèdica
Imatges per ressonància magnètica
Machine learning
Medical statistics
Magnetic resonance imaging
topic Aprenentatge automàtic
Estadística mèdica
Imatges per ressonància magnètica
Machine learning
Medical statistics
Magnetic resonance imaging
description Brain MRI researchers conducting multisite studies, such as within the ENIGMA Consortium, are very aware of the importance of controlling the effects of the site (EoS) in the statistical analysis. Conversely, authors of the novel machine-learning MRI studies may remove the EoS when training the machine-learning models but not control them when estimating the models' accuracy, potentially leading to severely biased estimates. We show examples from a toy simulation study and real MRI data in which we remove the EoS from both the "training set" and the "test set" during the training and application of the model. However, the accuracy is still inflated (or occasionally shrunk) unless we further control the EoS during the estimation of the accuracy. We also provide several methods for controlling the EoS during the estimation of the accuracy, and a simple R package ("multisite.accuracy") that smoothly does this task for several accuracy estimates (e.g.,sensitivity/specificity, area under the curve, correlation, hazard ratio, etc.).
publishDate 2021
dc.date.none.fl_str_mv 2021
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/acceptedVersion
format article
status_str acceptedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/2445/219882
url https://hdl.handle.net/2445/219882
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Versió postprint del document publicat a: https://doi.org/10.1016/j.pscychresns.2021.111313
Psychiatry Research-Neuroimaging, 2021, vol. 314
https://doi.org/10.1016/j.pscychresns.2021.111313
dc.rights.none.fl_str_mv cc-by-nc-nd (c) Elsevier B.V., 2021
http://creativecommons.org/licenses/by-nc-nd/4.0/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv cc-by-nc-nd (c) Elsevier B.V., 2021
http://creativecommons.org/licenses/by-nc-nd/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier B.V.
publisher.none.fl_str_mv Elsevier B.V.
dc.source.none.fl_str_mv Articles publicats en revistes (Medicina)
reponame:Dipòsit Digital de la UB
instname:Universidad de Barcelona
instname_str Universidad de Barcelona
reponame_str Dipòsit Digital de la UB
collection Dipòsit Digital de la UB
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869410405404639232
score 15,81155

Similares en LA Referencia