Modelling Uncertainty in Black-box Classification Systems

Programa de Doctorat en Matemàtica i Informàtica

Detalhes bibliográficos
Autor: Mena Roldán, José
Tipo de documento: tese
Estado:Versão publicada
Data de publicação:2020
País:España
Recursos:CBUC, CESCA
Repositório:TDR. Tesis Doctorales en Red
OAI Identifier:oai:www.tdx.cat:10803/670763
Acesso em linha:http://hdl.handle.net/10803/670763
Access Level:Acceso aberto
Palavra-chave:Sistemes classificadors (Intel·ligència artificial)
Sistemas clasificadores
Learning classifier systems
Incertesa (Teoria de la informació)
Incertidumbre (Teoría de la información)
Uncertainty (Information theory)
Aprenentatge automàtic
Aprendizaje automático
Machine learning
Ciències Experimentals i Matemàtiques
004
id ES_22c20c76e5a6dddeda6d5eb337dfeed8
oai_identifier_str oai:www.tdx.cat:10803/670763
network_acronym_str ES
network_name_str España
repository_id_str
spelling Modelling Uncertainty in Black-box Classification SystemsMena Roldán, JoséSistemes classificadors (Intel·ligència artificial)Sistemas clasificadoresLearning classifier systemsIncertesa (Teoria de la informació)Incertidumbre (Teoría de la información)Uncertainty (Information theory)Aprenentatge automàticAprendizaje automáticoMachine learningCiències Experimentals i Matemàtiques004Programa de Doctorat en Matemàtica i InformàticaCurrently, thanks to the Big Data boom, the excellent results obtained by deep learning models and the strong digital transformation experienced over the last years, many companies have decided to incorporate machine learning models into their systems. Some companies have detected this opportunity and are making a portfolio of artificial intelligence services available to third parties in the form of application programming interfaces (APIs). Subsequently, developers include calls to these APIs to incorporate AI functionalities in their products. Although it is an option that saves time and resources, it is true that, in most cases, these APIs are displayed in the form of blackboxes, the details of which are unknown to their clients. The complexity of such products typically leads to a lack of control and knowledge of the internal components, which, in turn, can drive to potential uncontrolled risks. Therefore, it is necessary to develop methods capable of evaluating the performance of these black-boxes when applied to a specific application. In this work, we present a robust uncertainty-based method for evaluating the performance of both probabilistic and categorical classification black-box models, in particular APIs, that enriches the predictions obtained with an uncertainty score. This uncertainty score enables the detection of inputs with very confident but erroneous predictions while protecting against out of distribution data points when deploying the model in a productive setting. In the first part of the thesis, we develop a thorough revision of the concept of uncertainty, focusing on the uncertainty of classification systems. We review the existingrelated literature, describing the different approaches for modelling this uncertainty, its application to different use cases and some of its desirable properties. Next, we introduce the proposed method for modelling uncertainty in black-box settings. Moreover, in the last chapters of the thesis, we showcase the method applied to different domains, including NLP and computer vision problems. Finally, we include two reallife applications of the method: classification of overqualification in job descriptions and readability assessment of texts.La tesis propone un método para el cálculo de la incertidumbre asociada a las predicciones de APIs o librerías externas de sistemas de clasificación.Universitat de BarcelonaVitrià i Marca, JordiPujol Vila, OriolTorrent Moreno, MarcUniversitat de Barcelona. Departament de Matemàtiques i Informàtica202120212020info:eu-repo/semantics/doctoralThesisinfo:eu-repo/semantics/publishedVersion148 p.application/pdfapplication/pdfhttp://hdl.handle.net/10803/670763TDX (Tesis Doctorals en Xarxa)reponame:TDR. Tesis Doctorales en Redinstname:CBUC, CESCAInglésL'accés als continguts d'aquesta tesi queda condicionat a l'acceptació de les condicions d'ús establertes per la següent llicència Creative Commons: http://creativecommons.org/licenses/by-nc/4.0/http://creativecommons.org/licenses/by-nc/4.0/info:eu-repo/semantics/openAccessoai:www.tdx.cat:10803/6707632026-06-14T12:46:07Z
dc.title.none.fl_str_mv Modelling Uncertainty in Black-box Classification Systems
title Modelling Uncertainty in Black-box Classification Systems
spellingShingle Modelling Uncertainty in Black-box Classification Systems
Mena Roldán, José
Sistemes classificadors (Intel·ligència artificial)
Sistemas clasificadores
Learning classifier systems
Incertesa (Teoria de la informació)
Incertidumbre (Teoría de la información)
Uncertainty (Information theory)
Aprenentatge automàtic
Aprendizaje automático
Machine learning
Ciències Experimentals i Matemàtiques
004
title_short Modelling Uncertainty in Black-box Classification Systems
title_full Modelling Uncertainty in Black-box Classification Systems
title_fullStr Modelling Uncertainty in Black-box Classification Systems
title_full_unstemmed Modelling Uncertainty in Black-box Classification Systems
title_sort Modelling Uncertainty in Black-box Classification Systems
dc.creator.none.fl_str_mv Mena Roldán, José
author Mena Roldán, José
author_facet Mena Roldán, José
author_role author
dc.contributor.none.fl_str_mv Vitrià i Marca, Jordi
Pujol Vila, Oriol
Torrent Moreno, Marc
Universitat de Barcelona. Departament de Matemàtiques i Informàtica
dc.subject.none.fl_str_mv Sistemes classificadors (Intel·ligència artificial)
Sistemas clasificadores
Learning classifier systems
Incertesa (Teoria de la informació)
Incertidumbre (Teoría de la información)
Uncertainty (Information theory)
Aprenentatge automàtic
Aprendizaje automático
Machine learning
Ciències Experimentals i Matemàtiques
004
topic Sistemes classificadors (Intel·ligència artificial)
Sistemas clasificadores
Learning classifier systems
Incertesa (Teoria de la informació)
Incertidumbre (Teoría de la información)
Uncertainty (Information theory)
Aprenentatge automàtic
Aprendizaje automático
Machine learning
Ciències Experimentals i Matemàtiques
004
description Programa de Doctorat en Matemàtica i Informàtica
publishDate 2020
dc.date.none.fl_str_mv 2020
2021
2021
dc.type.none.fl_str_mv info:eu-repo/semantics/doctoralThesis
info:eu-repo/semantics/publishedVersion
format doctoralThesis
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/10803/670763
url http://hdl.handle.net/10803/670763
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.rights.none.fl_str_mv http://creativecommons.org/licenses/by-nc/4.0/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv http://creativecommons.org/licenses/by-nc/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv 148 p.
application/pdf
application/pdf
dc.publisher.none.fl_str_mv Universitat de Barcelona
publisher.none.fl_str_mv Universitat de Barcelona
dc.source.none.fl_str_mv TDX (Tesis Doctorals en Xarxa)
reponame:TDR. Tesis Doctorales en Red
instname:CBUC, CESCA
instname_str CBUC, CESCA
reponame_str TDR. Tesis Doctorales en Red
collection TDR. Tesis Doctorales en Red
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869404600198496256
score 15,301603

Registros relacionados