Efficient deep ensembles by averaging neural networks in parameter space

Although deep ensembles provide large accuracy boosts relative to individual models, their use is not widespread in environments in which computational constraints are limited, as deep ensembles require storing M models and require M forward passes at prediction time. We propose a novel, computation...

ver descrição completa

Detalhes bibliográficos
Autor: Norris Mitchell, Philip
Tipo de documento: dissertação
Data de publicação:2021
País:España
Recursos:Universitat Politècnica de Catalunya (UPC)
Repositório:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglês
OAI Identifier:oai:upcommons.upc.edu:2117/356936
Acesso em linha:https://hdl.handle.net/2117/356936
Access Level:Acceso aberto
Palavra-chave:Artificial intelligence
Ensemble learning
Deep ensembles
Knowledge distillation
Permutation learning
Intel·ligència artificial
Classificació AMS::68 Computer science::68T Artificial intelligence
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
id ES_2cde2f2dae8efd9e6fc89516513d2169
oai_identifier_str oai:upcommons.upc.edu:2117/356936
network_acronym_str ES
network_name_str España
repository_id_str
spelling Efficient deep ensembles by averaging neural networks in parameter spaceNorris Mitchell, PhilipArtificial intelligenceEnsemble learningDeep ensemblesKnowledge distillationPermutation learningIntel·ligència artificialClassificació AMS::68 Computer science::68T Artificial intelligenceÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificialAlthough deep ensembles provide large accuracy boosts relative to individual models, their use is not widespread in environments in which computational constraints are limited, as deep ensembles require storing M models and require M forward passes at prediction time. We propose a novel, computationally efficient alternative, which we name permAVG. Although deep ensembles cannot simply be average in parameter space, as all models find distinct perhaps distant local optima, permAVG exploits the symmetries of the loss landscape by learning permutations, such that all M models can be permuted into the same local optimum and can thereafter safely be averaged.Universitat Politècnica de CatalunyaAgudo Martínez, AntonioRuiz Ovejero, Adrià20212021-10-0120212021-11-23master thesishttp://purl.org/coar/resource_type/c_bdccNAhttp://purl.org/coar/version/c_be7fb7dd8ff6fe43info:eu-repo/semantics/masterThesisapplication/pdfhttps://hdl.handle.net/2117/356936reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/3569362026-05-27T15:37:01Z
dc.title.none.fl_str_mv Efficient deep ensembles by averaging neural networks in parameter space
title Efficient deep ensembles by averaging neural networks in parameter space
spellingShingle Efficient deep ensembles by averaging neural networks in parameter space
Norris Mitchell, Philip
Artificial intelligence
Ensemble learning
Deep ensembles
Knowledge distillation
Permutation learning
Intel·ligència artificial
Classificació AMS::68 Computer science::68T Artificial intelligence
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
title_short Efficient deep ensembles by averaging neural networks in parameter space
title_full Efficient deep ensembles by averaging neural networks in parameter space
title_fullStr Efficient deep ensembles by averaging neural networks in parameter space
title_full_unstemmed Efficient deep ensembles by averaging neural networks in parameter space
title_sort Efficient deep ensembles by averaging neural networks in parameter space
dc.creator.none.fl_str_mv Norris Mitchell, Philip
author Norris Mitchell, Philip
author_facet Norris Mitchell, Philip
author_role author
dc.contributor.none.fl_str_mv Agudo Martínez, Antonio
Ruiz Ovejero, Adrià
dc.subject.none.fl_str_mv Artificial intelligence
Ensemble learning
Deep ensembles
Knowledge distillation
Permutation learning
Intel·ligència artificial
Classificació AMS::68 Computer science::68T Artificial intelligence
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
topic Artificial intelligence
Ensemble learning
Deep ensembles
Knowledge distillation
Permutation learning
Intel·ligència artificial
Classificació AMS::68 Computer science::68T Artificial intelligence
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
description Although deep ensembles provide large accuracy boosts relative to individual models, their use is not widespread in environments in which computational constraints are limited, as deep ensembles require storing M models and require M forward passes at prediction time. We propose a novel, computationally efficient alternative, which we name permAVG. Although deep ensembles cannot simply be average in parameter space, as all models find distinct perhaps distant local optima, permAVG exploits the symmetries of the loss landscape by learning permutations, such that all M models can be permuted into the same local optimum and can thereafter safely be averaged.
publishDate 2021
dc.date.none.fl_str_mv 2021
2021-10-01
2021
2021-11-23
dc.type.none.fl_str_mv master thesis
http://purl.org/coar/resource_type/c_bdcc
NA
http://purl.org/coar/version/c_be7fb7dd8ff6fe43
dc.type.openaire.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/356936
url https://hdl.handle.net/2117/356936
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Universitat Politècnica de Catalunya
publisher.none.fl_str_mv Universitat Politècnica de Catalunya
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869405271059595264
score 15,300719

Registros relacionados