Adversarial α-divergence minimization for Bayesian approximate inference

Neural networks are state-of-the-art models for machine learning problems. They are often trained via back-propagation to find a value of the weights that correctly predicts the observed data. Back-propagation has shown good performance in many applications, however, it cannot easily output an estim...

Descripción completa

Detalles Bibliográficos
Autores: Rodríguez Santana, Simón, Hernández Lobato, Daniel
Tipo de recurso: artículo
Fecha de publicación:2020
País:España
Institución:Universidad Autónoma de Madrid
Repositorio:Biblos-e Archivo. Repositorio Institucional de la UAM
Idioma:inglés
OAI Identifier:oai:repositorio.uam.es:10486/711383
Acceso en línea:http://hdl.handle.net/10486/711383
https://dx.doi.org/10.1016/j.neucom.2020.09.076
Access Level:acceso abierto
Palabra clave:Bayesian Neural Networks
Approximate Inference
Alpha Divergences
Adversarial Variational Bayes
Informática
id ES_a6c54ae5ceb280ea7b96f34cc7a482f1
oai_identifier_str oai:repositorio.uam.es:10486/711383
network_acronym_str ES
network_name_str España
repository_id_str
spelling Adversarial α-divergence minimization for Bayesian approximate inferenceRodríguez Santana, SimónHernández Lobato, DanielBayesian Neural NetworksApproximate InferenceAlpha DivergencesAdversarial Variational BayesInformáticaNeural networks are state-of-the-art models for machine learning problems. They are often trained via back-propagation to find a value of the weights that correctly predicts the observed data. Back-propagation has shown good performance in many applications, however, it cannot easily output an estimate of the uncertainty in the predictions made. Estimating this uncertainty is a critical aspect with important applications. One method to obtain this information consists in following a Bayesian approach to obtain a posterior distribution of the model parameters. This posterior distribution summarizes which parameter values are compatible with the observed data. However, the posterior is often intractable and has to be approximated. Several methods have been devised for this task. Here, we propose a general method for approximate Bayesian inference that is based on minimizing -divergences, and that allows for flexible approximate distributions. We call this method adversarial -divergence minimization (AADM). We have evaluated AADM in the context of Bayesian neural networks. Extensive experiments show that it may lead to better results in terms of the test log-likelihood, and sometimes in terms of the squared error, in regression problems. In classification problems, however, AADM gives competitive results.Simón Rodríguez acknowledges the Spanish Ministry of Economy for the FPI SEV-2015–0554-16–4 Ph.D. grant. The authors gratefully acknowledge the use of the facilities of Centro de Computación Científica (CCC) at Universidad Autónoma de Madrid. Daniel Hernández-Lobato also acknowledges financial support from PID2019-106827 GB-I00/ AEI/ 10.13039/501100011033 and from Spanish Plan Nacional I+D+i, grant TIN2016-76406-P.ElsevierDepartamento de Ingeniería InformáticaEscuela Politécnica Superior20202020-11-06research articlehttp://purl.org/coar/resource_type/c_2df8fbb1AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/711383https://dx.doi.org/10.1016/j.neucom.2020.09.076reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/7113832026-06-23T12:46:27Z
dc.title.none.fl_str_mv Adversarial α-divergence minimization for Bayesian approximate inference
title Adversarial α-divergence minimization for Bayesian approximate inference
spellingShingle Adversarial α-divergence minimization for Bayesian approximate inference
Rodríguez Santana, Simón
Bayesian Neural Networks
Approximate Inference
Alpha Divergences
Adversarial Variational Bayes
Informática
title_short Adversarial α-divergence minimization for Bayesian approximate inference
title_full Adversarial α-divergence minimization for Bayesian approximate inference
title_fullStr Adversarial α-divergence minimization for Bayesian approximate inference
title_full_unstemmed Adversarial α-divergence minimization for Bayesian approximate inference
title_sort Adversarial α-divergence minimization for Bayesian approximate inference
dc.creator.none.fl_str_mv Rodríguez Santana, Simón
Hernández Lobato, Daniel
author Rodríguez Santana, Simón
author_facet Rodríguez Santana, Simón
Hernández Lobato, Daniel
author_role author
author2 Hernández Lobato, Daniel
author2_role author
dc.contributor.none.fl_str_mv Departamento de Ingeniería Informática
Escuela Politécnica Superior
dc.subject.none.fl_str_mv Bayesian Neural Networks
Approximate Inference
Alpha Divergences
Adversarial Variational Bayes
Informática
topic Bayesian Neural Networks
Approximate Inference
Alpha Divergences
Adversarial Variational Bayes
Informática
description Neural networks are state-of-the-art models for machine learning problems. They are often trained via back-propagation to find a value of the weights that correctly predicts the observed data. Back-propagation has shown good performance in many applications, however, it cannot easily output an estimate of the uncertainty in the predictions made. Estimating this uncertainty is a critical aspect with important applications. One method to obtain this information consists in following a Bayesian approach to obtain a posterior distribution of the model parameters. This posterior distribution summarizes which parameter values are compatible with the observed data. However, the posterior is often intractable and has to be approximated. Several methods have been devised for this task. Here, we propose a general method for approximate Bayesian inference that is based on minimizing -divergences, and that allows for flexible approximate distributions. We call this method adversarial -divergence minimization (AADM). We have evaluated AADM in the context of Bayesian neural networks. Extensive experiments show that it may lead to better results in terms of the test log-likelihood, and sometimes in terms of the squared error, in regression problems. In classification problems, however, AADM gives competitive results.
publishDate 2020
dc.date.none.fl_str_mv 2020
2020-11-06
dc.type.none.fl_str_mv research article
http://purl.org/coar/resource_type/c_2df8fbb1
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv http://hdl.handle.net/10486/711383
https://dx.doi.org/10.1016/j.neucom.2020.09.076
url http://hdl.handle.net/10486/711383
https://dx.doi.org/10.1016/j.neucom.2020.09.076
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:Biblos-e Archivo. Repositorio Institucional de la UAM
instname:Universidad Autónoma de Madrid
instname_str Universidad Autónoma de Madrid
reponame_str Biblos-e Archivo. Repositorio Institucional de la UAM
collection Biblos-e Archivo. Repositorio Institucional de la UAM
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869415724606291968
score 15,301603

Similares en LA Referencia