Meta-learning for neural network weight prediction and compressive learning
The rapid expansion in the size of new datasets and available data online has led to significant scaling of the size of neural models. However, training deep learning models and performing hyperparameter tuning is computationally demanding and very time-consuming. Thus, there is a growing need for f...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2023 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/403830 |
| Acceso en línea: | https://hdl.handle.net/2117/403830 |
| Access Level: | acceso embargado |
| Palabra clave: | Deep learning (Machine learning) Machine learning Genomics meta-learning deep learning machine learning hypernetwork tabular data genomics compressive learning pca regression classification Aprenentatge profund Aprenentatge automàtic Genòmica Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial |
| Sumario: | The rapid expansion in the size of new datasets and available data online has led to significant scaling of the size of neural models. However, training deep learning models and performing hyperparameter tuning is computationally demanding and very time-consuming. Thus, there is a growing need for fast and efficient learning techniques. In this thesis, we explore two meta-learning approaches to provide faster and data-driven systems for multiple tasks. First, we introduce HyperFast, a meta-trained hypernetwork designed for instant classification of tabular data in a single forward pass. HyperFast generates the weights of a task-specific neural network tailored to an unseen dataset that can be directly used for classification inference, removing the need for training a model. We report extensive experiments with OpenML and genomic data, comparing HyperFast to competing tabular data neural networks, traditional ML methods, AutoML systems, and boosting machines such as XGBoost. HyperFast shows highly competitive results, while being significantly faster. Our approach introduces a promising paradigm for fast classification and rapid model deployment. Second, we propose a Compressive Meta-Learning framework as an alternative to traditional compressive learning, wherein random, non-linear features are used to project large-scale databases onto compact information-preserving representations. These database-level summaries are later used to learn parameters of interest from the underlying data distribution, without the need to access the original samples, providing an efficient and privacy-friendly learning framework. However, both the encoding and decoding techniques are typically randomized and data-independent, not benefiting from the underlying structure of the data. In our approach, we meta-learn both the encoding and decoding stages of compressive learning methods in a data-driven fashion by using neural networks. To showcase the potential of the presented Compressive Meta-Learning framework, we show results for neural network-based compressive principal component analysis (PCA) and compressive ridge regression. |
|---|