Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models

Earth system models (ESMs) are intricate models employed for simulating the Earth's climate, typically constructed from distinct independent components dedicated to simulate specific natural phenomena (such as atmosphere and ocean dynamics, atmospheric chemistry, land and ocean biosphere). In o...

Descripción completa

Detalles Bibliográficos
Autores: Palomas Martínez, Sergi, Acosta Cobos, Mario César, Utrera Iglesias, Gladys Miriam|||0000-0002-0637-1297, Tourigny, Etienne|||0000-0003-4628-1461
Tipo de recurso: artículo
Fecha de publicación:2025
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/439285
Acceso en línea:https://hdl.handle.net/2117/439285
https://dx.doi.org/10.5194/gmd-18-3661-2025
Access Level:acceso abierto
Palabra clave:Earth system models
ESMs
Climate modeling
High-performance computing
HPC
Automatic load-balance
Resource allocation
Fittingness
Àrees temàtiques de la UPC::Informàtica::Aplicacions de la informàtica::Aplicacions informàtiques a la física i l‘enginyeria
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
id ES_f8521afcffaf4530288fcfdc4ef15e54
oai_identifier_str oai:upcommons.upc.edu:2117/439285
network_acronym_str ES
network_name_str España
repository_id_str
dc.title.none.fl_str_mv Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
title Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
spellingShingle Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
Palomas Martínez, Sergi
Earth system models
ESMs
Climate modeling
High-performance computing
HPC
Automatic load-balance
Resource allocation
Fittingness
Àrees temàtiques de la UPC::Informàtica::Aplicacions de la informàtica::Aplicacions informàtiques a la física i l‘enginyeria
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
title_short Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
title_full Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
title_fullStr Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
title_full_unstemmed Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
title_sort Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system models
dc.creator.none.fl_str_mv Palomas Martínez, Sergi
Acosta Cobos, Mario César
Utrera Iglesias, Gladys Miriam|||0000-0002-0637-1297
Tourigny, Etienne|||0000-0003-4628-1461
author Palomas Martínez, Sergi
author_facet Palomas Martínez, Sergi
Acosta Cobos, Mario César
Utrera Iglesias, Gladys Miriam|||0000-0002-0637-1297
Tourigny, Etienne|||0000-0003-4628-1461
author_role author
author2 Acosta Cobos, Mario César
Utrera Iglesias, Gladys Miriam|||0000-0002-0637-1297
Tourigny, Etienne|||0000-0003-4628-1461
author2_role author
author
author
dc.subject.none.fl_str_mv Earth system models
ESMs
Climate modeling
High-performance computing
HPC
Automatic load-balance
Resource allocation
Fittingness
Àrees temàtiques de la UPC::Informàtica::Aplicacions de la informàtica::Aplicacions informàtiques a la física i l‘enginyeria
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
topic Earth system models
ESMs
Climate modeling
High-performance computing
HPC
Automatic load-balance
Resource allocation
Fittingness
Àrees temàtiques de la UPC::Informàtica::Aplicacions de la informàtica::Aplicacions informàtiques a la física i l‘enginyeria
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
description Earth system models (ESMs) are intricate models employed for simulating the Earth's climate, typically constructed from distinct independent components dedicated to simulate specific natural phenomena (such as atmosphere and ocean dynamics, atmospheric chemistry, land and ocean biosphere). In order to capture the interactions between these processes, ESMs utilize coupling libraries, which oversee the synchronization and field exchanges among independent developed codes typically operating in parallel as a multi-program, multiple data (MPMD) application. The performance achieved depends on the coupling approach, as well as on the number of parallel resources and scalability properties of each component. Determining the appropriate number of resources to use for each component in coupled ESMs is crucial for efficient utilization of the high-performance computing (HPC) infrastructures used in climate modeling. However, this task traditionally involves manual testing of multiple process allocations by trial and error, requiring significant time investment from researchers and making the process more error-prone, often resulting in a loss of application performance due to the complexity of the task. This paper introduces the automatic load-balance tool (auto-lb), a methodology and tool for determining the resource allocation to each component within coupled ESMs, aimed at improving the application's performance. Notably, this methodology is automatic and does not require expertise in HPC to improve the performance achieved by coupled ESMs. This is accomplished by minimizing the load imbalance: reducing each constituent's execution cost (core hours), as well as minimizing the core hours wasted resulting from the synchronizations between them, without penalizing the execution speed of the entire model. This optimization is achieved regardless of the scalability properties of each constituent and the complexity of their dependencies during the coupling. To achieve this, we designed a new performance metric called “fittingness” to assess the performance of coupled execution, evaluating the trade-off between parallel efficiency and application throughput. This metric is intended for scenarios where optimality can depend on various criteria and constraints. Aiming for maximum speed might not be desirable if it leads to a decrease in parallel efficiency and thus increases the computational costs of simulation. The methodology was tested across multiple experiments using the widely recognized European ESM, EC-Earth3. The results were compared with real operational configurations, such as those used for the Coupled Model Intercomparison Project Phase 6 (CMIP6) and European Climate Prediction project (EUCP), and validated on different HPC platforms. All of them suggest that the current approaches lead to performance loss, and that auto-lb can achieve better results in both execution speed and reduction of the core hours needed. When comparing to the EC-Earth standard-resolution CPMIP6 runs, we achieved a configuration 4.7 % faster while also reducing the core hours required by 1.3 %. Likewise, when compared to the EC-Earth high-resolution EUCP runs, the method presented showed an improvement of 34 % in the speed, with a 6.7 % reduction in the core hours consumed.
publishDate 2025
dc.date.none.fl_str_mv 2025
2025-06-20
2025
2025-07-24
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/439285
https://dx.doi.org/10.5194/gmd-18-3661-2025
url https://hdl.handle.net/2117/439285
https://dx.doi.org/10.5194/gmd-18-3661-2025
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv European Commission http://doi.org/10.13039/501100000780 HE 101093054 Center of excellence for weather and climate phase 3
European Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 824084 Infrastructure for the European Network for Earth System modelling
Agencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2020-116324RA-I00 OPTIMIZACION DE MODELOS DEL SISTEMA TERRESTRE EN EL CAMINO A LA NUEVA GENERACION DE SISTEMAS "EXASCALE" DE ALTO RENDIMIENTO
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv European Geosciences Union (EGU)
publisher.none.fl_str_mv European Geosciences Union (EGU)
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869424991021301760
spelling Reducing time and computing costs in EC-Earth: an automatic load-balancing approach for coupled Earth system modelsPalomas Martínez, SergiAcosta Cobos, Mario CésarUtrera Iglesias, Gladys Miriam|||0000-0002-0637-1297Tourigny, Etienne|||0000-0003-4628-1461Earth system modelsESMsClimate modelingHigh-performance computingHPCAutomatic load-balanceResource allocationFittingnessÀrees temàtiques de la UPC::Informàtica::Aplicacions de la informàtica::Aplicacions informàtiques a la física i l‘enginyeriaÀrees temàtiques de la UPC::Informàtica::Arquitectura de computadorsEarth system models (ESMs) are intricate models employed for simulating the Earth's climate, typically constructed from distinct independent components dedicated to simulate specific natural phenomena (such as atmosphere and ocean dynamics, atmospheric chemistry, land and ocean biosphere). In order to capture the interactions between these processes, ESMs utilize coupling libraries, which oversee the synchronization and field exchanges among independent developed codes typically operating in parallel as a multi-program, multiple data (MPMD) application. The performance achieved depends on the coupling approach, as well as on the number of parallel resources and scalability properties of each component. Determining the appropriate number of resources to use for each component in coupled ESMs is crucial for efficient utilization of the high-performance computing (HPC) infrastructures used in climate modeling. However, this task traditionally involves manual testing of multiple process allocations by trial and error, requiring significant time investment from researchers and making the process more error-prone, often resulting in a loss of application performance due to the complexity of the task. This paper introduces the automatic load-balance tool (auto-lb), a methodology and tool for determining the resource allocation to each component within coupled ESMs, aimed at improving the application's performance. Notably, this methodology is automatic and does not require expertise in HPC to improve the performance achieved by coupled ESMs. This is accomplished by minimizing the load imbalance: reducing each constituent's execution cost (core hours), as well as minimizing the core hours wasted resulting from the synchronizations between them, without penalizing the execution speed of the entire model. This optimization is achieved regardless of the scalability properties of each constituent and the complexity of their dependencies during the coupling. To achieve this, we designed a new performance metric called “fittingness” to assess the performance of coupled execution, evaluating the trade-off between parallel efficiency and application throughput. This metric is intended for scenarios where optimality can depend on various criteria and constraints. Aiming for maximum speed might not be desirable if it leads to a decrease in parallel efficiency and thus increases the computational costs of simulation. The methodology was tested across multiple experiments using the widely recognized European ESM, EC-Earth3. The results were compared with real operational configurations, such as those used for the Coupled Model Intercomparison Project Phase 6 (CMIP6) and European Climate Prediction project (EUCP), and validated on different HPC platforms. All of them suggest that the current approaches lead to performance loss, and that auto-lb can achieve better results in both execution speed and reduction of the core hours needed. When comparing to the EC-Earth standard-resolution CPMIP6 runs, we achieved a configuration 4.7 % faster while also reducing the core hours required by 1.3 %. Likewise, when compared to the EC-Earth high-resolution EUCP runs, the method presented showed an improvement of 34 % in the speed, with a 6.7 % reduction in the core hours consumed.This research was supported by the European High Performance Computing Joint Undertaking (EuroHPC JU) and the European Union (EU) through the ESiWACE3 project under grant agreement no. 101093054, as well as the ISENES3 project under grant agreement no. H2020-GA-824084. Additional funding was provided by the National Research Agency through the OEMES project (grant no. PID2020-116324RA-I00).Peer ReviewedEuropean Geosciences Union (EGU)20252025-06-2020252025-07-24journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/439285https://dx.doi.org/10.5194/gmd-18-3661-2025reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengEuropean Commission http://doi.org/10.13039/501100000780 HE 101093054 Center of excellence for weather and climate phase 3European Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 824084 Infrastructure for the European Network for Earth System modellingAgencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2020-116324RA-I00 OPTIMIZACION DE MODELOS DEL SISTEMA TERRESTRE EN EL CAMINO A LA NUEVA GENERACION DE SISTEMAS "EXASCALE" DE ALTO RENDIMIENTOopen accesshttp://purl.org/coar/access_right/c_abf2Attribution 4.0 Internationalhttp://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4392852026-05-27T15:37:01Z
score 15.811543

Similares en LA Referencia