Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms

Real-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. For example, minimizing both an estimate of the generalization error of a deep neural network and its prediction time....

ver descrição completa

Detalhes bibliográficos
Autores: Garrido Merchán, Eduardo César, Fernández Sánchez, Daniel, Hernández Lobato, Daniel
Formato: artículo
Fecha de publicación:2023
País:España
Recursos:Universidad Autónoma de Madrid
Repositorio:Biblos-e Archivo. Repositorio Institucional de la UAM
Idioma:inglés
OAI Identifier:oai:repositorio.uam.es:10486/709939
Acesso em linha:http://hdl.handle.net/10486/709939
https://dx.doi.org/10.1016/j.eswa.2022.119328
Access Level:acceso abierto
Palavra-chave:Bayesian optimization
Gaussian processes
Batch optimization
Constrained multi-objective optimization
Informática
id ES_f07613bce87b4f16f48b6da0283f9e99
oai_identifier_str oai:repositorio.uam.es:10486/709939
network_acronym_str ES
network_name_str España
repository_id_str
spelling Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithmsGarrido Merchán, Eduardo CésarFernández Sánchez, DanielHernández Lobato, DanielBayesian optimizationGaussian processesBatch optimizationConstrained multi-objective optimizationInformáticaReal-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. For example, minimizing both an estimate of the generalization error of a deep neural network and its prediction time. We may also consider, as a constraint, that the deep neural network must be implemented in a chip with an area below some size. Here, both the objectives and the constraint are black boxes, i.e., functions whose analytical expressions are unknown and are expensive to evaluate. Bayesian optimization (BO) methods have shown state-of-the-art results in these tasks. For this, they evaluate iteratively, at carefully chosen locations, the objectives and the constraints with the goal of solving the optimization problem in a small number of iterations. Nevertheless, most BO methods are sequential and perform evaluations at just one input location, at each iteration. Sometimes, however, we may evaluate several configurations in parallel. If this is the case, as when a cluster of computers is available, sequential evaluations result in a waste of resources. To avoid this, one has to choose which locations to evaluate in parallel, at each iteration. This article introduces PPESMOC, Parallel Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints, an information based batch method for the simultaneous optimization of multiple expensive-to-evaluate black-box functions under the presence of several constraints. Iteratively, PPESMOC selects a batch of input locations at which to evaluate the black-boxes in parallel so as to maximally reduce the entropy of the Pareto set of the optimization problem. To our knowledge, this is the first information-based batch method for constrained multi-objective BO. We present empirical evidence in the form of several optimization problems that illustrate the effectiveness of PPESMOC. Moreover, we also show in several experiments the utility of the proposed method to tune the hyper-parameters of machine learning algorithms.Authors gratefully acknowledge the use of the facilities of Centro de Computacion Cientifica (CCC) at Universidad Autónoma de Madrid. The authors also acknowledge financial support from Spanish Plan Nacional I+D+i, PID2019-106827GB-I00 /AEI / 10.13039/5011000- 11033ElsevierDepartamento de Ingeniería InformáticaEscuela Politécnica Superior20232023-04-01research articlehttp://purl.org/coar/resource_type/c_2df8fbb1AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/709939https://dx.doi.org/10.1016/j.eswa.2022.119328reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/7099392026-06-23T12:46:27Z
dc.title.none.fl_str_mv Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
title Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
spellingShingle Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
Garrido Merchán, Eduardo César
Bayesian optimization
Gaussian processes
Batch optimization
Constrained multi-objective optimization
Informática
title_short Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
title_full Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
title_fullStr Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
title_full_unstemmed Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
title_sort Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms
dc.creator.none.fl_str_mv Garrido Merchán, Eduardo César
Fernández Sánchez, Daniel
Hernández Lobato, Daniel
author Garrido Merchán, Eduardo César
author_facet Garrido Merchán, Eduardo César
Fernández Sánchez, Daniel
Hernández Lobato, Daniel
author_role author
author2 Fernández Sánchez, Daniel
Hernández Lobato, Daniel
author2_role author
author
dc.contributor.none.fl_str_mv Departamento de Ingeniería Informática
Escuela Politécnica Superior
dc.subject.none.fl_str_mv Bayesian optimization
Gaussian processes
Batch optimization
Constrained multi-objective optimization
Informática
topic Bayesian optimization
Gaussian processes
Batch optimization
Constrained multi-objective optimization
Informática
description Real-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. For example, minimizing both an estimate of the generalization error of a deep neural network and its prediction time. We may also consider, as a constraint, that the deep neural network must be implemented in a chip with an area below some size. Here, both the objectives and the constraint are black boxes, i.e., functions whose analytical expressions are unknown and are expensive to evaluate. Bayesian optimization (BO) methods have shown state-of-the-art results in these tasks. For this, they evaluate iteratively, at carefully chosen locations, the objectives and the constraints with the goal of solving the optimization problem in a small number of iterations. Nevertheless, most BO methods are sequential and perform evaluations at just one input location, at each iteration. Sometimes, however, we may evaluate several configurations in parallel. If this is the case, as when a cluster of computers is available, sequential evaluations result in a waste of resources. To avoid this, one has to choose which locations to evaluate in parallel, at each iteration. This article introduces PPESMOC, Parallel Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints, an information based batch method for the simultaneous optimization of multiple expensive-to-evaluate black-box functions under the presence of several constraints. Iteratively, PPESMOC selects a batch of input locations at which to evaluate the black-boxes in parallel so as to maximally reduce the entropy of the Pareto set of the optimization problem. To our knowledge, this is the first information-based batch method for constrained multi-objective BO. We present empirical evidence in the form of several optimization problems that illustrate the effectiveness of PPESMOC. Moreover, we also show in several experiments the utility of the proposed method to tune the hyper-parameters of machine learning algorithms.
publishDate 2023
dc.date.none.fl_str_mv 2023
2023-04-01
dc.type.none.fl_str_mv research article
http://purl.org/coar/resource_type/c_2df8fbb1
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv http://hdl.handle.net/10486/709939
https://dx.doi.org/10.1016/j.eswa.2022.119328
url http://hdl.handle.net/10486/709939
https://dx.doi.org/10.1016/j.eswa.2022.119328
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:Biblos-e Archivo. Repositorio Institucional de la UAM
instname:Universidad Autónoma de Madrid
instname_str Universidad Autónoma de Madrid
reponame_str Biblos-e Archivo. Repositorio Institucional de la UAM
collection Biblos-e Archivo. Repositorio Institucional de la UAM
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869423958224273408
score 15,811543

Registos relacionados