Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs

Convolutional neural networks (CNNs) have recently attracted considerable attention due to their outstanding accuracy in applications, such as image recognition and natural language processing. While one advantage of the CNNs over other types of neural networks is their reduced computational cost, f...

Descripción completa

Detalles Bibliográficos
Autores: Jordà, Marc, Valero-Lara, Pedro, Peña Monferrer, Antonio José
Tipo de recurso: artículo
Fecha de publicación:2019
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/167384
Acceso en línea:https://hdl.handle.net/2117/167384
https://dx.doi.org/10.1109/ACCESS.2019.2918851
Access Level:acceso abierto
Palabra clave:High performance computing
Neural network
Convolution
Deep learning
cuDNN
GPU
Volta
Supercomputadors
Àrees temàtiques de la UPC::Informàtica
id ES_b3e83e38f2c6c4bbf0db915b79b2f0bf
oai_identifier_str oai:upcommons.upc.edu:2117/167384
network_acronym_str ES
network_name_str España
repository_id_str
spelling Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUsJordà, MarcValero-Lara, PedroPeña Monferrer, Antonio JoséHigh performance computingNeural networkConvolutionDeep learningcuDNNGPUVoltaSupercomputadorsÀrees temàtiques de la UPC::InformàticaConvolutional neural networks (CNNs) have recently attracted considerable attention due to their outstanding accuracy in applications, such as image recognition and natural language processing. While one advantage of the CNNs over other types of neural networks is their reduced computational cost, faster execution is still desired for both training and inference. Since convolution operations pose most of the execution time, multiple algorithms were and are being developed with the aim of accelerating this type of operations. However, due to the wide range of convolution parameter configurations used in the CNNs and the possible data type representations, it is not straightforward to assess in advance which of the available algorithms will be the best performing in each particular case. In this paper, we present a performance evaluation of the convolution algorithms provided by the cuDNN, the library used by most deep learning frameworks for their GPU operations. In our analysis, we leverage the convolution parameter configurations from widely used the CNNs and discuss which algorithms are better suited depending on the convolution parameters for both 32 and 16-bit floating-point (FP) data representations. Our results show that the filter size and the number of inputs are the most significant parameters when selecting a GPU convolution algorithm for 32-bit FP data. For 16-bit FP, leveraging specialized arithmetic units (NVIDIA Tensor Cores) is key to obtain the best performance.This work was supported by the European Union's Horizon 2020 Research and Innovation Program under the Marie Sklodowska-Curie under Grant 749516, and in part by the Spanish Juan de la Cierva under Grant IJCI-2017-33511Peer ReviewedIEEE20192019-05-2420192019-08-02journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/167384https://dx.doi.org/10.1109/ACCESS.2019.2918851reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengEuropean Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 749516 Advanced Ecosystem for Broad Heterogeneous Memory Usageopen accesshttp://purl.org/coar/access_right/c_abf2Attribution-NonCommercial-NoDerivs 3.0 Spainhttp://creativecommons.org/licenses/by-nc-nd/3.0/es/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/1673842026-05-27T15:37:01Z
dc.title.none.fl_str_mv Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
title Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
spellingShingle Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
Jordà, Marc
High performance computing
Neural network
Convolution
Deep learning
cuDNN
GPU
Volta
Supercomputadors
Àrees temàtiques de la UPC::Informàtica
title_short Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
title_full Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
title_fullStr Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
title_full_unstemmed Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
title_sort Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
dc.creator.none.fl_str_mv Jordà, Marc
Valero-Lara, Pedro
Peña Monferrer, Antonio José
author Jordà, Marc
author_facet Jordà, Marc
Valero-Lara, Pedro
Peña Monferrer, Antonio José
author_role author
author2 Valero-Lara, Pedro
Peña Monferrer, Antonio José
author2_role author
author
dc.subject.none.fl_str_mv High performance computing
Neural network
Convolution
Deep learning
cuDNN
GPU
Volta
Supercomputadors
Àrees temàtiques de la UPC::Informàtica
topic High performance computing
Neural network
Convolution
Deep learning
cuDNN
GPU
Volta
Supercomputadors
Àrees temàtiques de la UPC::Informàtica
description Convolutional neural networks (CNNs) have recently attracted considerable attention due to their outstanding accuracy in applications, such as image recognition and natural language processing. While one advantage of the CNNs over other types of neural networks is their reduced computational cost, faster execution is still desired for both training and inference. Since convolution operations pose most of the execution time, multiple algorithms were and are being developed with the aim of accelerating this type of operations. However, due to the wide range of convolution parameter configurations used in the CNNs and the possible data type representations, it is not straightforward to assess in advance which of the available algorithms will be the best performing in each particular case. In this paper, we present a performance evaluation of the convolution algorithms provided by the cuDNN, the library used by most deep learning frameworks for their GPU operations. In our analysis, we leverage the convolution parameter configurations from widely used the CNNs and discuss which algorithms are better suited depending on the convolution parameters for both 32 and 16-bit floating-point (FP) data representations. Our results show that the filter size and the number of inputs are the most significant parameters when selecting a GPU convolution algorithm for 32-bit FP data. For 16-bit FP, leveraging specialized arithmetic units (NVIDIA Tensor Cores) is key to obtain the best performance.
publishDate 2019
dc.date.none.fl_str_mv 2019
2019-05-24
2019
2019-08-02
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/167384
https://dx.doi.org/10.1109/ACCESS.2019.2918851
url https://hdl.handle.net/2117/167384
https://dx.doi.org/10.1109/ACCESS.2019.2918851
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv European Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 749516 Advanced Ecosystem for Broad Heterogeneous Memory Usage
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivs 3.0 Spain
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivs 3.0 Spain
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv IEEE
publisher.none.fl_str_mv IEEE
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869417216637665280
score 15.300724

Similares en LA Referencia