Multicore workflow characterisation methodology for payloads running in the ALICE Grid

For LHC Run3 the ALICE experiment software stack has been completely refactored, incorporating support for multicore job execution. Whereas in both LHC Run 1 and 2 the Grid jobs were single-process and made use of a single CPU core, the new multicore jobs spawn multiple processes and threads within...

ver descrição completa

Detalhes bibliográficos
Autores: Bertran Ferrer, Marta, Grigoras, Costin, Badia Sala, Rosa Maria|||0000-0003-2941-5499
Formato: artículo
Fecha de publicación:2024
País:España
Recursos:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/409320
Acesso em linha:https://hdl.handle.net/2117/409320
https://dx.doi.org/10.1051/epjconf/202429504005
Access Level:acceso abierto
Palavra-chave:Resource allocation
Computational grids (Computer systems)
Assignació de recursos
Computació distribuïda
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors::Arquitectures distribuïdes
id ES_c717cdbb11b6a3a84ff4f5fa0da42847
oai_identifier_str oai:upcommons.upc.edu:2117/409320
network_acronym_str ES
network_name_str España
repository_id_str
spelling Multicore workflow characterisation methodology for payloads running in the ALICE GridBertran Ferrer, MartaGrigoras, CostinBadia Sala, Rosa Maria|||0000-0003-2941-5499Resource allocationComputational grids (Computer systems)Assignació de recursosComputació distribuïdaÀrees temàtiques de la UPC::Informàtica::Arquitectura de computadors::Arquitectures distribuïdesFor LHC Run3 the ALICE experiment software stack has been completely refactored, incorporating support for multicore job execution. Whereas in both LHC Run 1 and 2 the Grid jobs were single-process and made use of a single CPU core, the new multicore jobs spawn multiple processes and threads within the payload. Some of these multicore jobs deploy a high amount of shortlived processes, in the order of more than a dozen per second. The overhead of starting so many processes impacts the overall CPU utilization of the payloads, in particular its System component. Furthermore, the short-lived processes were not correctly accounted for by the monitoring system of the experiment. This paper presents the developed new methodology for supervising the payload execution. We also present a black box analysis of the new multicore experiment software framework tracing the used resources and system function calls issued by MonteCarlo simulation jobs. Multiple sources of overhead in the lifecycle of processes and threads have thus been identified. This paper describes how the source of each was traced and what solutions were implemented to address them. These improvements have impacted the resource consumption and the overall turnaround time of these payloads with a notable 35% reduction in execution time for a reference production job. We also introduce how this methodology will be used to further improve the efficiency of our experiment software and what other optimization venues are currently under research.Peer ReviewedEPJ Web of Conferences20242024-05-0620242024-06-06journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/409320https://dx.doi.org/10.1051/epjconf/202429504005reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2Attribution 4.0 Internationalhttp://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4093202026-05-27T15:37:01Z
dc.title.none.fl_str_mv Multicore workflow characterisation methodology for payloads running in the ALICE Grid
title Multicore workflow characterisation methodology for payloads running in the ALICE Grid
spellingShingle Multicore workflow characterisation methodology for payloads running in the ALICE Grid
Bertran Ferrer, Marta
Resource allocation
Computational grids (Computer systems)
Assignació de recursos
Computació distribuïda
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors::Arquitectures distribuïdes
title_short Multicore workflow characterisation methodology for payloads running in the ALICE Grid
title_full Multicore workflow characterisation methodology for payloads running in the ALICE Grid
title_fullStr Multicore workflow characterisation methodology for payloads running in the ALICE Grid
title_full_unstemmed Multicore workflow characterisation methodology for payloads running in the ALICE Grid
title_sort Multicore workflow characterisation methodology for payloads running in the ALICE Grid
dc.creator.none.fl_str_mv Bertran Ferrer, Marta
Grigoras, Costin
Badia Sala, Rosa Maria|||0000-0003-2941-5499
author Bertran Ferrer, Marta
author_facet Bertran Ferrer, Marta
Grigoras, Costin
Badia Sala, Rosa Maria|||0000-0003-2941-5499
author_role author
author2 Grigoras, Costin
Badia Sala, Rosa Maria|||0000-0003-2941-5499
author2_role author
author
dc.subject.none.fl_str_mv Resource allocation
Computational grids (Computer systems)
Assignació de recursos
Computació distribuïda
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors::Arquitectures distribuïdes
topic Resource allocation
Computational grids (Computer systems)
Assignació de recursos
Computació distribuïda
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors::Arquitectures distribuïdes
description For LHC Run3 the ALICE experiment software stack has been completely refactored, incorporating support for multicore job execution. Whereas in both LHC Run 1 and 2 the Grid jobs were single-process and made use of a single CPU core, the new multicore jobs spawn multiple processes and threads within the payload. Some of these multicore jobs deploy a high amount of shortlived processes, in the order of more than a dozen per second. The overhead of starting so many processes impacts the overall CPU utilization of the payloads, in particular its System component. Furthermore, the short-lived processes were not correctly accounted for by the monitoring system of the experiment. This paper presents the developed new methodology for supervising the payload execution. We also present a black box analysis of the new multicore experiment software framework tracing the used resources and system function calls issued by MonteCarlo simulation jobs. Multiple sources of overhead in the lifecycle of processes and threads have thus been identified. This paper describes how the source of each was traced and what solutions were implemented to address them. These improvements have impacted the resource consumption and the overall turnaround time of these payloads with a notable 35% reduction in execution time for a reference production job. We also introduce how this methodology will be used to further improve the efficiency of our experiment software and what other optimization venues are currently under research.
publishDate 2024
dc.date.none.fl_str_mv 2024
2024-05-06
2024
2024-06-06
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/409320
https://dx.doi.org/10.1051/epjconf/202429504005
url https://hdl.handle.net/2117/409320
https://dx.doi.org/10.1051/epjconf/202429504005
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv EPJ Web of Conferences
publisher.none.fl_str_mv EPJ Web of Conferences
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869419128014503936
score 15,81155

Registos relacionados