Ideal step size estimation for the multinomial logistic regression
At the core of deep learning optimization problems reside algorithms such as the Stochastic Gradient Descent (SGD), which employs a subset of the data per iteration to estimate the gradient in order to minimize a cost function. Adaptive algorithms, based on SGD, are well known for being effective in...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2024 |
| País: | Perú |
| Institución: | Pontificia Universidad Católica del Perú |
| Repositorio: | PUCP-Tesis |
| Idioma: | inglés |
| OAI Identifier: | oai:tesis.pucp.edu.pe:20.500.12404/29791 |
| Acceso en línea: | http://hdl.handle.net/20.500.12404/29791 |
| Access Level: | acceso abierto |
| Palabra clave: | Aprendizaje automático (Inteligencia artificial) Aprendizaje profundo (Aprendizaje automático) Optimización matemática Análisis de regresión https://purl.org/pe-repo/ocde/ford#2.00.00 |
| Sumario: | At the core of deep learning optimization problems reside algorithms such as the Stochastic Gradient Descent (SGD), which employs a subset of the data per iteration to estimate the gradient in order to minimize a cost function. Adaptive algorithms, based on SGD, are well known for being effective in using gradient information from past iterations, generating momentum or memory that enables a more accurate prediction of the true gradient slope in future iterations, thus accelerating convergence. Nevertheless, these algorithms still need an initial (scalar) learning rate (LR) as well as a LR scheduler. In this work we propose a new SGD algorithm that estimates the initial (scalar) LR via an adaptation of the ideal Cauchy step size for the multinomial logistic regression; furthermore, the LR is recursively updated up to a given number of epochs, after which a decaying LR scheduler is used. The proposed method is assessed for several well-known multiclass classification architectures and favorably compares against other well-tuned (scalar and spatially) adaptive alternatives, including the Adam algorithm. |
|---|