Alpha-divergence minimization for deep Gaussian processes
This paper proposes the minimization of α-divergences for approximate inference in the context of deep Gaussian processes (DGPs). The proposed method can be considered as a generalization of variational inference (VI) and expectation propagation (EP), two previously used methods for approximate infe...
| Autores: | , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2022 |
| País: | España |
| Institución: | Universidad Autónoma de Madrid |
| Repositorio: | Biblos-e Archivo. Repositorio Institucional de la UAM |
| Idioma: | inglés |
| OAI Identifier: | oai:repositorio.uam.es:10486/704028 |
| Acceso en línea: | http://hdl.handle.net/10486/704028 https://dx.doi.org/10.1016/j.ijar.2022.08.003 |
| Access Level: | acceso abierto |
| Palabra clave: | Deep Gaussian processes Expectation propagation α-divergences Approximate inference Variational inference Informática |
| id |
ES_303571ea061343da4dfbcd7faac8705e |
|---|---|
| oai_identifier_str |
oai:repositorio.uam.es:10486/704028 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| spelling |
Alpha-divergence minimization for deep Gaussian processesVillacampa Calvo, CarlosHernández Muñoz, GonzaloHernández Lobato, DanielDeep Gaussian processesExpectation propagationα-divergencesApproximate inferenceVariational inferenceInformáticaThis paper proposes the minimization of α-divergences for approximate inference in the context of deep Gaussian processes (DGPs). The proposed method can be considered as a generalization of variational inference (VI) and expectation propagation (EP), two previously used methods for approximate inference in DGPs. Both VI and EP are based on the minimization of the Kullback-Leibler divergence. The proposed method is based on a scalable version of power expectation propagation, a method that introduces an extra parameter α that specifies the targeted α-divergence to be optimized. In particular, such a method can recover the VI solution when α → 0 and the EP solution when α → 1. An exhaustive experimental evaluation shows that the minimization of α-divergences via the proposed method is feasible in DGPs and that choosing intermediate values of the α parameter between 0 and 1 can give better results in some problems. This means that one can improve the results of VI and EP when training DGPs. Importantly, the proposed method allows for stochastic optimization techniques, making it able to address datasets with several millions of instancesThe authors gratefully acknowledge the use of the facilities of Centro de Computación Científica (CCC) at Universidad Autónoma de Madrid. The authors also acknowledge financial support from Spanish Plan Nacional I+D+i, Ministerio de Ciencia e Innovación, grant PID2019-106827GB-I00 / AEI / 10.13039/501100011033ElsevierDepartamento de Ingeniería InformáticaEscuela Politécnica Superior20222022-08-22research articlehttp://purl.org/coar/resource_type/c_2df8fbb1VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/704028https://dx.doi.org/10.1016/j.ijar.2022.08.003reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/7040282026-06-23T12:46:27Z |
| dc.title.none.fl_str_mv |
Alpha-divergence minimization for deep Gaussian processes |
| title |
Alpha-divergence minimization for deep Gaussian processes |
| spellingShingle |
Alpha-divergence minimization for deep Gaussian processes Villacampa Calvo, Carlos Deep Gaussian processes Expectation propagation α-divergences Approximate inference Variational inference Informática |
| title_short |
Alpha-divergence minimization for deep Gaussian processes |
| title_full |
Alpha-divergence minimization for deep Gaussian processes |
| title_fullStr |
Alpha-divergence minimization for deep Gaussian processes |
| title_full_unstemmed |
Alpha-divergence minimization for deep Gaussian processes |
| title_sort |
Alpha-divergence minimization for deep Gaussian processes |
| dc.creator.none.fl_str_mv |
Villacampa Calvo, Carlos Hernández Muñoz, Gonzalo Hernández Lobato, Daniel |
| author |
Villacampa Calvo, Carlos |
| author_facet |
Villacampa Calvo, Carlos Hernández Muñoz, Gonzalo Hernández Lobato, Daniel |
| author_role |
author |
| author2 |
Hernández Muñoz, Gonzalo Hernández Lobato, Daniel |
| author2_role |
author author |
| dc.contributor.none.fl_str_mv |
Departamento de Ingeniería Informática Escuela Politécnica Superior |
| dc.subject.none.fl_str_mv |
Deep Gaussian processes Expectation propagation α-divergences Approximate inference Variational inference Informática |
| topic |
Deep Gaussian processes Expectation propagation α-divergences Approximate inference Variational inference Informática |
| description |
This paper proposes the minimization of α-divergences for approximate inference in the context of deep Gaussian processes (DGPs). The proposed method can be considered as a generalization of variational inference (VI) and expectation propagation (EP), two previously used methods for approximate inference in DGPs. Both VI and EP are based on the minimization of the Kullback-Leibler divergence. The proposed method is based on a scalable version of power expectation propagation, a method that introduces an extra parameter α that specifies the targeted α-divergence to be optimized. In particular, such a method can recover the VI solution when α → 0 and the EP solution when α → 1. An exhaustive experimental evaluation shows that the minimization of α-divergences via the proposed method is feasible in DGPs and that choosing intermediate values of the α parameter between 0 and 1 can give better results in some problems. This means that one can improve the results of VI and EP when training DGPs. Importantly, the proposed method allows for stochastic optimization techniques, making it able to address datasets with several millions of instances |
| publishDate |
2022 |
| dc.date.none.fl_str_mv |
2022 2022-08-22 |
| dc.type.none.fl_str_mv |
research article http://purl.org/coar/resource_type/c_2df8fbb1 VoR http://purl.org/coar/version/c_970fb48d4fbd8a85 |
| dc.type.openaire.fl_str_mv |
info:eu-repo/semantics/article |
| format |
article |
| dc.identifier.none.fl_str_mv |
http://hdl.handle.net/10486/704028 https://dx.doi.org/10.1016/j.ijar.2022.08.003 |
| url |
http://hdl.handle.net/10486/704028 https://dx.doi.org/10.1016/j.ijar.2022.08.003 |
| dc.language.none.fl_str_mv |
Inglés eng |
| language_invalid_str_mv |
Inglés |
| language |
eng |
| dc.rights.none.fl_str_mv |
open access http://purl.org/coar/access_right/c_abf2 |
| dc.rights.openaire.fl_str_mv |
info:eu-repo/semantics/openAccess |
| rights_invalid_str_mv |
open access http://purl.org/coar/access_right/c_abf2 |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf |
| dc.publisher.none.fl_str_mv |
Elsevier |
| publisher.none.fl_str_mv |
Elsevier |
| dc.source.none.fl_str_mv |
reponame:Biblos-e Archivo. Repositorio Institucional de la UAM instname:Universidad Autónoma de Madrid |
| instname_str |
Universidad Autónoma de Madrid |
| reponame_str |
Biblos-e Archivo. Repositorio Institucional de la UAM |
| collection |
Biblos-e Archivo. Repositorio Institucional de la UAM |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869405530086178816 |
| score |
15,300724 |