Joint outlier detection and variable selection using discrete optimization
In regression, the quality of estimators is known to be very sensitive to the presence of spurious variables and outliers. Unfortunately, this is a frequent situation when dealing with real data. To handle outlier proneness and achieve variable selection, we propose a robust method performing the ou...
| Autores: | , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2021 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/362112 |
| Acceso en línea: | https://hdl.handle.net/2117/362112 https://dx.doi.org/10.2436/20.8080.02.109 |
| Access Level: | acceso abierto |
| Palabra clave: | Robust optimization statistical learning linear regression variable selection outlier detection mixed integer programming Programació (Matemàtica) Intel·ligència artificial Estadística matemàtica Classificació AMS::62 Statistics::62J Linear inference, regression Classificació AMS::62 Statistics::62G Nonparametric inference Classificació AMS::68 Computer science::68T Artificial intelligence Classificació AMS::90 Operations research, mathematical programming::90C Mathematical programming Àrees temàtiques de la UPC::Matemàtiques i estadística::Estadística matemàtica |
| id |
ES_abde978ffb5a1e8807ea970a8113e25a |
|---|---|
| oai_identifier_str |
oai:upcommons.upc.edu:2117/362112 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| spelling |
Joint outlier detection and variable selection using discrete optimizationJammal, MahdiCanu, StephaneAbdallah, MaherRobust optimizationstatistical learninglinear regressionvariable selectionoutlier detectionmixed integer programmingProgramació (Matemàtica)Intel·ligència artificialEstadística matemàticaClassificació AMS::62 Statistics::62J Linear inference, regressionClassificació AMS::62 Statistics::62G Nonparametric inferenceClassificació AMS::68 Computer science::68T Artificial intelligenceClassificació AMS::90 Operations research, mathematical programming::90C Mathematical programmingÀrees temàtiques de la UPC::Matemàtiques i estadística::Estadística matemàticaIn regression, the quality of estimators is known to be very sensitive to the presence of spurious variables and outliers. Unfortunately, this is a frequent situation when dealing with real data. To handle outlier proneness and achieve variable selection, we propose a robust method performing the outright rejection of discordant observations together with the selection of relevant variables. A natural way to define the corresponding optimization problem is to use the ℓ0 norm and recast it as a mixed integer optimization problem. To retrieve this global solution more efficiently, we suggest the use of additional constraints as well as a clever initialization. To this end, an efficient and scalable non-convex proximal alternate algorithm is introduced. An empirical comparison between the ℓ0 norm approach and its ℓ1 relaxation is presented as well. Results on both synthetic and real data sets provided that the mixed integer programming approach and its discrete first order warm start provide high quality solutions.Peer ReviewedInstitut d'Estadística de Catalunya20212021-06-1520222022-02-10journal articlehttp://purl.org/coar/resource_type/c_6501NAhttp://purl.org/coar/version/c_be7fb7dd8ff6fe43info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/362112https://dx.doi.org/10.2436/20.8080.02.109reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2Attribution-NonCommercial-NoDerivs 3.0 Spainhttp://creativecommons.org/licenses/by-nc-nd/3.0/es/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/3621122026-05-27T15:37:01Z |
| dc.title.none.fl_str_mv |
Joint outlier detection and variable selection using discrete optimization |
| title |
Joint outlier detection and variable selection using discrete optimization |
| spellingShingle |
Joint outlier detection and variable selection using discrete optimization Jammal, Mahdi Robust optimization statistical learning linear regression variable selection outlier detection mixed integer programming Programació (Matemàtica) Intel·ligència artificial Estadística matemàtica Classificació AMS::62 Statistics::62J Linear inference, regression Classificació AMS::62 Statistics::62G Nonparametric inference Classificació AMS::68 Computer science::68T Artificial intelligence Classificació AMS::90 Operations research, mathematical programming::90C Mathematical programming Àrees temàtiques de la UPC::Matemàtiques i estadística::Estadística matemàtica |
| title_short |
Joint outlier detection and variable selection using discrete optimization |
| title_full |
Joint outlier detection and variable selection using discrete optimization |
| title_fullStr |
Joint outlier detection and variable selection using discrete optimization |
| title_full_unstemmed |
Joint outlier detection and variable selection using discrete optimization |
| title_sort |
Joint outlier detection and variable selection using discrete optimization |
| dc.creator.none.fl_str_mv |
Jammal, Mahdi Canu, Stephane Abdallah, Maher |
| author |
Jammal, Mahdi |
| author_facet |
Jammal, Mahdi Canu, Stephane Abdallah, Maher |
| author_role |
author |
| author2 |
Canu, Stephane Abdallah, Maher |
| author2_role |
author author |
| dc.subject.none.fl_str_mv |
Robust optimization statistical learning linear regression variable selection outlier detection mixed integer programming Programació (Matemàtica) Intel·ligència artificial Estadística matemàtica Classificació AMS::62 Statistics::62J Linear inference, regression Classificació AMS::62 Statistics::62G Nonparametric inference Classificació AMS::68 Computer science::68T Artificial intelligence Classificació AMS::90 Operations research, mathematical programming::90C Mathematical programming Àrees temàtiques de la UPC::Matemàtiques i estadística::Estadística matemàtica |
| topic |
Robust optimization statistical learning linear regression variable selection outlier detection mixed integer programming Programació (Matemàtica) Intel·ligència artificial Estadística matemàtica Classificació AMS::62 Statistics::62J Linear inference, regression Classificació AMS::62 Statistics::62G Nonparametric inference Classificació AMS::68 Computer science::68T Artificial intelligence Classificació AMS::90 Operations research, mathematical programming::90C Mathematical programming Àrees temàtiques de la UPC::Matemàtiques i estadística::Estadística matemàtica |
| description |
In regression, the quality of estimators is known to be very sensitive to the presence of spurious variables and outliers. Unfortunately, this is a frequent situation when dealing with real data. To handle outlier proneness and achieve variable selection, we propose a robust method performing the outright rejection of discordant observations together with the selection of relevant variables. A natural way to define the corresponding optimization problem is to use the ℓ0 norm and recast it as a mixed integer optimization problem. To retrieve this global solution more efficiently, we suggest the use of additional constraints as well as a clever initialization. To this end, an efficient and scalable non-convex proximal alternate algorithm is introduced. An empirical comparison between the ℓ0 norm approach and its ℓ1 relaxation is presented as well. Results on both synthetic and real data sets provided that the mixed integer programming approach and its discrete first order warm start provide high quality solutions. |
| publishDate |
2021 |
| dc.date.none.fl_str_mv |
2021 2021-06-15 2022 2022-02-10 |
| dc.type.none.fl_str_mv |
journal article http://purl.org/coar/resource_type/c_6501 NA http://purl.org/coar/version/c_be7fb7dd8ff6fe43 |
| dc.type.openaire.fl_str_mv |
info:eu-repo/semantics/article |
| format |
article |
| dc.identifier.none.fl_str_mv |
https://hdl.handle.net/2117/362112 https://dx.doi.org/10.2436/20.8080.02.109 |
| url |
https://hdl.handle.net/2117/362112 https://dx.doi.org/10.2436/20.8080.02.109 |
| dc.language.none.fl_str_mv |
Inglés eng |
| language_invalid_str_mv |
Inglés |
| language |
eng |
| dc.rights.none.fl_str_mv |
open access http://purl.org/coar/access_right/c_abf2 Attribution-NonCommercial-NoDerivs 3.0 Spain http://creativecommons.org/licenses/by-nc-nd/3.0/es/ |
| dc.rights.openaire.fl_str_mv |
info:eu-repo/semantics/openAccess |
| rights_invalid_str_mv |
open access http://purl.org/coar/access_right/c_abf2 Attribution-NonCommercial-NoDerivs 3.0 Spain http://creativecommons.org/licenses/by-nc-nd/3.0/es/ |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf |
| dc.publisher.none.fl_str_mv |
Institut d'Estadística de Catalunya |
| publisher.none.fl_str_mv |
Institut d'Estadística de Catalunya |
| dc.source.none.fl_str_mv |
reponame:UPCommons. Portal del coneixement obert de la UPC instname:Universitat Politècnica de Catalunya (UPC) |
| instname_str |
Universitat Politècnica de Catalunya (UPC) |
| reponame_str |
UPCommons. Portal del coneixement obert de la UPC |
| collection |
UPCommons. Portal del coneixement obert de la UPC |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869416306607915008 |
| score |
15.301603 |