A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes

Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next...

ver descrição completa

Detalhes bibliográficos
Autores: Badosa, Ferrán, Espinosa, Antonio, Acevedo, Cesar, Vera, Gonzalo, Ripoll, Ana
Tipo de documento: artigo
Estado:Versão publicada
Data de publicação:2019
País:España
Recursos:Consejo Superior de Investigaciones Científicas (CSIC)
Repositório:DIGITAL.CSIC. Repositorio Institucional del CSIC
OAI Identifier:oai:dnet:digitalcsic_::9798e7394789f84b5a9661f67a87d3e4
Acesso em linha:http://hdl.handle.net/10261/207022
Access Level:Acceso aberto
Palavra-chave:Resource manager
Bioinformatics workflows
Multivariate regression prediction
Scheduling algorithm
id ES_c8efbe666df98b36195fa640738caafe
oai_identifier_str oai:dnet:digitalcsic_::9798e7394789f84b5a9661f67a87d3e4
network_acronym_str ES
network_name_str España
repository_id_str
spelling A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodesBadosa, FerránEspinosa, AntonioAcevedo, CesarVera, GonzaloRipoll, AnaResource managerBioinformatics workflowsMultivariate regression predictionScheduling algorithmBioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next. Resource management systems in clusters don’t consider these variations and subsequent needs. As a result, the computing power offered by clusters is not harnessed properly, compromising both application performance and resource efficiency. To tackle these issues, we propose a History-Based Resource Manager for bioinformatics workflows applications running on clusters with heterogeneous nodes. The proposed resource manager features a prediction model that generates multiple performance predictions for each job under different combinations of cluster resources. Furthermore, the proposed resource manager includes a scheduling algorithm that considers the degree of multiprogramming of the nodes, scheduling combinations of applications for simultaneous same-node execution upon their compatibility. To test the proposed resource manager, we process two workloads formed by different amounts of workflows made up by common bioinformatics applications. Results prove that for the given cases, the proposed resource manager improves the performance obtained with SLURM, using First Come First Served policy. The proposal shows an average workflow makespan improvement range between 28 and 35%, averaging 32%, an average workflow efficiency improvement range between 75 and 83%, averaging 79%, and an average resource usage improvement range between 96 and 101%, averaging 99%. Furthermore, the proposed scheduling algorithm can improve the average workflow makespan by a range of values between 26 and 36%, averaging 31%, compared to Max–Min and Min–Min algorithms.Peer reviewedSpringer NatureBadosa, Ferran [0000-0002-9162-2918]Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]202020202019info:eu-repo/semantics/articlehttp://purl.org/coar/resource_type/c_6501Publisher's versioninfo:eu-repo/semantics/publishedVersionhttp://hdl.handle.net/10261/207022reponame:DIGITAL.CSIC. Repositorio Institucional del CSICinstname:Consejo Superior de Investigaciones Científicas (CSIC)Ingléshttps://doi.org/10.1007/s10766-018-0600-zSíinfo:eu-repo/semantics/openAccessoai:dnet:digitalcsic_::9798e7394789f84b5a9661f67a87d3e42026-05-22T06:33:51Z
dc.title.none.fl_str_mv A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
title A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
spellingShingle A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
Badosa, Ferrán
Resource manager
Bioinformatics workflows
Multivariate regression prediction
Scheduling algorithm
title_short A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
title_full A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
title_fullStr A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
title_full_unstemmed A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
title_sort A history-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
dc.creator.none.fl_str_mv Badosa, Ferrán
Espinosa, Antonio
Acevedo, Cesar
Vera, Gonzalo
Ripoll, Ana
author Badosa, Ferrán
author_facet Badosa, Ferrán
Espinosa, Antonio
Acevedo, Cesar
Vera, Gonzalo
Ripoll, Ana
author_role author
author2 Espinosa, Antonio
Acevedo, Cesar
Vera, Gonzalo
Ripoll, Ana
author2_role author
author
author
author
dc.contributor.none.fl_str_mv Badosa, Ferran [0000-0002-9162-2918]
Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]
dc.subject.none.fl_str_mv Resource manager
Bioinformatics workflows
Multivariate regression prediction
Scheduling algorithm
topic Resource manager
Bioinformatics workflows
Multivariate regression prediction
Scheduling algorithm
description Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next. Resource management systems in clusters don’t consider these variations and subsequent needs. As a result, the computing power offered by clusters is not harnessed properly, compromising both application performance and resource efficiency. To tackle these issues, we propose a History-Based Resource Manager for bioinformatics workflows applications running on clusters with heterogeneous nodes. The proposed resource manager features a prediction model that generates multiple performance predictions for each job under different combinations of cluster resources. Furthermore, the proposed resource manager includes a scheduling algorithm that considers the degree of multiprogramming of the nodes, scheduling combinations of applications for simultaneous same-node execution upon their compatibility. To test the proposed resource manager, we process two workloads formed by different amounts of workflows made up by common bioinformatics applications. Results prove that for the given cases, the proposed resource manager improves the performance obtained with SLURM, using First Come First Served policy. The proposal shows an average workflow makespan improvement range between 28 and 35%, averaging 32%, an average workflow efficiency improvement range between 75 and 83%, averaging 79%, and an average resource usage improvement range between 96 and 101%, averaging 99%. Furthermore, the proposed scheduling algorithm can improve the average workflow makespan by a range of values between 26 and 36%, averaging 31%, compared to Max–Min and Min–Min algorithms.
publishDate 2019
dc.date.none.fl_str_mv 2019
2020
2020
dc.type.none.fl_str_mv info:eu-repo/semantics/article
http://purl.org/coar/resource_type/c_6501
Publisher's version
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/10261/207022
url http://hdl.handle.net/10261/207022
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv https://doi.org/10.1007/s10766-018-0600-z
Sí
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.publisher.none.fl_str_mv Springer Nature
publisher.none.fl_str_mv Springer Nature
dc.source.none.fl_str_mv reponame:DIGITAL.CSIC. Repositorio Institucional del CSIC
instname:Consejo Superior de Investigaciones Científicas (CSIC)
instname_str Consejo Superior de Investigaciones Científicas (CSIC)
reponame_str DIGITAL.CSIC. Repositorio Institucional del CSIC
collection DIGITAL.CSIC. Repositorio Institucional del CSIC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869419328337608704
score 15,812429

Registros relacionados