Heterogeneous gradient computing optimization for scalable deep neural networks

Nowadays, data processing applications based on neural networks cope with the growth in the amount of data to be processed and with the increase in both the depth and complexity of the neural networks architectures, and hence in the number of parameters to be learned. High-performance computing plat...

Full description

Bibliographic Details
Authors: Moreno Álvarez, Sergio, Paoletti, Mercedes Eugenia, Rico Gallego, Juan Antonio, Haut, Juan M.
Format: article
Publication Date:2022
Country:España
Institution:Universidad Nacional de Educación a Distancia
Repository:e-spacio. Repositorio Institucional de la UNED
Language:English
OAI Identifier:oai:e-spacio.uned.es:20.500.14468/24403
Online Access:https://hdl.handle.net/20.500.14468/24403
Access Level:Open access
Keyword:12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
deep learning
deep neural networks
high-performance computing
heterogeneous platforms
distributed training
id ES_8f991006a38d161facecf282907a9ae0
oai_identifier_str oai:e-spacio.uned.es:20.500.14468/24403
network_acronym_str ES
network_name_str España
repository_id_str
spelling Heterogeneous gradient computing optimization for scalable deep neural networksMoreno Álvarez, SergioPaoletti, Mercedes EugeniaRico Gallego, Juan AntonioHaut, Juan M.12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informáticadeep learningdeep neural networkshigh-performance computingheterogeneous platformsdistributed trainingNowadays, data processing applications based on neural networks cope with the growth in the amount of data to be processed and with the increase in both the depth and complexity of the neural networks architectures, and hence in the number of parameters to be learned. High-performance computing platforms are provided with fast computing resources, including multi-core processors and graphical processing units, to manage such computational burden of deep neural network applications. A common optimization technique is to distribute the workload between the processes deployed on the resources of the platform. This approach is known as data-parallelism. Each process, known as replica, trains its own copy of the model on a disjoint data partition. Nevertheless, the heterogeneity of the computational resources composing the platform requires to unevenly distribute the workload between the replicas according to its computational capabilities, to optimize the overall execution performance. Since the amount of data to be processed is different in each replica, the influence of the gradients computed by the replicas in the global parameter updating should be different. This work proposes a modification of the gradient computation method that considers the different speeds of the replicas, and hence, its amount of data assigned. The experimental results have been conducted on heterogeneous high-performance computing platforms for a wide range of models and datasets, showing an improvement in the final accuracy with respect to current techniques, with a comparable performance.Springerhttps://orcid.org/0000-0003-1030-3729https://orcid.org/0000-0002-4264-7473https://orcid.org/0000-0001-6701-961Xe-Spacio UNED20242024-11-1820222022-01-0120222022-01-01journal articlehttp://purl.org/coar/resource_type/c_6501info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/20.500.14468/24403reponame:e-spacio. Repositorio Institucional de la UNEDinstname:Universidad Nacional de Educación a DistanciaInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccesshttp://creativecommons.org/licenses/by-nc-nd/4.0/deed.esoai:e-spacio.uned.es:20.500.14468/244032026-06-06T12:38:31Z
dc.title.none.fl_str_mv Heterogeneous gradient computing optimization for scalable deep neural networks
title Heterogeneous gradient computing optimization for scalable deep neural networks
spellingShingle Heterogeneous gradient computing optimization for scalable deep neural networks
Moreno Álvarez, Sergio
12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
deep learning
deep neural networks
high-performance computing
heterogeneous platforms
distributed training
title_short Heterogeneous gradient computing optimization for scalable deep neural networks
title_full Heterogeneous gradient computing optimization for scalable deep neural networks
title_fullStr Heterogeneous gradient computing optimization for scalable deep neural networks
title_full_unstemmed Heterogeneous gradient computing optimization for scalable deep neural networks
title_sort Heterogeneous gradient computing optimization for scalable deep neural networks
dc.creator.none.fl_str_mv Moreno Álvarez, Sergio
Paoletti, Mercedes Eugenia
Rico Gallego, Juan Antonio
Haut, Juan M.
author Moreno Álvarez, Sergio
author_facet Moreno Álvarez, Sergio
Paoletti, Mercedes Eugenia
Rico Gallego, Juan Antonio
Haut, Juan M.
author_role author
author2 Paoletti, Mercedes Eugenia
Rico Gallego, Juan Antonio
Haut, Juan M.
author2_role author
author
author
dc.contributor.none.fl_str_mv https://orcid.org/0000-0003-1030-3729
https://orcid.org/0000-0002-4264-7473
https://orcid.org/0000-0001-6701-961X
e-Spacio UNED
dc.subject.none.fl_str_mv 12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
deep learning
deep neural networks
high-performance computing
heterogeneous platforms
distributed training
topic 12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
deep learning
deep neural networks
high-performance computing
heterogeneous platforms
distributed training
description Nowadays, data processing applications based on neural networks cope with the growth in the amount of data to be processed and with the increase in both the depth and complexity of the neural networks architectures, and hence in the number of parameters to be learned. High-performance computing platforms are provided with fast computing resources, including multi-core processors and graphical processing units, to manage such computational burden of deep neural network applications. A common optimization technique is to distribute the workload between the processes deployed on the resources of the platform. This approach is known as data-parallelism. Each process, known as replica, trains its own copy of the model on a disjoint data partition. Nevertheless, the heterogeneity of the computational resources composing the platform requires to unevenly distribute the workload between the replicas according to its computational capabilities, to optimize the overall execution performance. Since the amount of data to be processed is different in each replica, the influence of the gradients computed by the replicas in the global parameter updating should be different. This work proposes a modification of the gradient computation method that considers the different speeds of the replicas, and hence, its amount of data assigned. The experimental results have been conducted on heterogeneous high-performance computing platforms for a wide range of models and datasets, showing an improvement in the final accuracy with respect to current techniques, with a comparable performance.
publishDate 2022
dc.date.none.fl_str_mv 2022
2022-01-01
2022
2022-01-01
2024
2024-11-18
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/20.500.14468/24403
url https://hdl.handle.net/20.500.14468/24403
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
info:eu-repo/semantics/openAccess
http://creativecommons.org/licenses/by-nc-nd/4.0/deed.es
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
http://creativecommons.org/licenses/by-nc-nd/4.0/deed.es
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Springer
publisher.none.fl_str_mv Springer
dc.source.none.fl_str_mv reponame:e-spacio. Repositorio Institucional de la UNED
instname:Universidad Nacional de Educación a Distancia
instname_str Universidad Nacional de Educación a Distancia
reponame_str e-spacio. Repositorio Institucional de la UNED
collection e-spacio. Repositorio Institucional de la UNED
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869413225396699136
score 15.812429

Similar Items