The impact of class imbalance in classification performance metrics based on the binary confusion matrix

A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain othe...

Full description

Bibliographic Details
Authors: Luque Sendra, Amalia, Carrasco Muñoz, Alejandro, Martín-Gómez, Alejandro Manuel, Heras García de Vinuesa, Ana de las
Format: article
Status:Published version
Publication Date:2019
Country:España
Institution:Universidad de Sevilla (US)
Repository:idUS. Depósito de Investigación de la Universidad de Sevilla
OAI Identifier:oai:idus.us.es:11441/85586
Online Access:https://hdl.handle.net/11441/85586
https://doi.org/10.1016/j.patcog.2019.02.023
Access Level:Open access
Keyword:Classification
Performance measures
Imbalanced datasets
Class Balance Metrics
id ES_ecdca840fc991bf97f0d051183afce01
oai_identifier_str oai:idus.us.es:11441/85586
network_acronym_str ES
network_name_str España
repository_id_str
spelling The impact of class imbalance in classification performance metrics based on the binary confusion matrixLuque Sendra, AmaliaCarrasco Muñoz, AlejandroMartín-Gómez, Alejandro ManuelHeras García de Vinuesa, Ana de lasClassificationPerformance measuresImbalanced datasetsClass Balance MetricsA major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other well-known performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of null-biased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics.Elsevier ScienceIngeniería del DiseñoTecnología Electrónica2019info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttps://hdl.handle.net/11441/85586https://doi.org/10.1016/j.patcog.2019.02.023reponame:idUS. Depósito de Investigación de la Universidad de Sevillainstname:Universidad de Sevilla (US)InglésPattern Recognition, 91, 216-231.https://doi.org/10.1016/j.patcog.2019.02.023info:eu-repo/semantics/openAccessoai:idus.us.es:11441/855862026-06-17T12:51:07Z
dc.title.none.fl_str_mv The impact of class imbalance in classification performance metrics based on the binary confusion matrix
title The impact of class imbalance in classification performance metrics based on the binary confusion matrix
spellingShingle The impact of class imbalance in classification performance metrics based on the binary confusion matrix
Luque Sendra, Amalia
Classification
Performance measures
Imbalanced datasets
Class Balance Metrics
title_short The impact of class imbalance in classification performance metrics based on the binary confusion matrix
title_full The impact of class imbalance in classification performance metrics based on the binary confusion matrix
title_fullStr The impact of class imbalance in classification performance metrics based on the binary confusion matrix
title_full_unstemmed The impact of class imbalance in classification performance metrics based on the binary confusion matrix
title_sort The impact of class imbalance in classification performance metrics based on the binary confusion matrix
dc.creator.none.fl_str_mv Luque Sendra, Amalia
Carrasco Muñoz, Alejandro
Martín-Gómez, Alejandro Manuel
Heras García de Vinuesa, Ana de las
author Luque Sendra, Amalia
author_facet Luque Sendra, Amalia
Carrasco Muñoz, Alejandro
Martín-Gómez, Alejandro Manuel
Heras García de Vinuesa, Ana de las
author_role author
author2 Carrasco Muñoz, Alejandro
Martín-Gómez, Alejandro Manuel
Heras García de Vinuesa, Ana de las
author2_role author
author
author
dc.contributor.none.fl_str_mv Ingeniería del Diseño
Tecnología Electrónica
dc.subject.none.fl_str_mv Classification
Performance measures
Imbalanced datasets
Class Balance Metrics
topic Classification
Performance measures
Imbalanced datasets
Class Balance Metrics
description A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other well-known performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of null-biased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics.
publishDate 2019
dc.date.none.fl_str_mv 2019
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/11441/85586
https://doi.org/10.1016/j.patcog.2019.02.023
url https://hdl.handle.net/11441/85586
https://doi.org/10.1016/j.patcog.2019.02.023
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Pattern Recognition, 91, 216-231.
https://doi.org/10.1016/j.patcog.2019.02.023
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Elsevier Science
publisher.none.fl_str_mv Elsevier Science
dc.source.none.fl_str_mv reponame:idUS. Depósito de Investigación de la Universidad de Sevilla
instname:Universidad de Sevilla (US)
instname_str Universidad de Sevilla (US)
reponame_str idUS. Depósito de Investigación de la Universidad de Sevilla
collection idUS. Depósito de Investigación de la Universidad de Sevilla
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869423376882204672
score 15,301603

Similar Items