Statistical analysis of the performance of four Apache Spark ML Algorithms
Feature selection (FS) techniques generally require repeatedly training and evaluating models to assess the importance of each feature for a particular task. However, due to the increasing size of currently available databases, distributed processing has become a necessity for many tasks. In this co...
| Autores: | , , , |
|---|---|
| Tipo de recurso: | artículo |
| Estado: | Versión publicada |
| Fecha de publicación: | 2022 |
| País: | Argentina |
| Institución: | Universidad Nacional de La Plata |
| Repositorio: | SEDICI (UNLP) |
| Idioma: | inglés |
| OAI Identifier: | oai:sedici.unlp.edu.ar:10915/146934 |
| Acceso en línea: | http://sedici.unlp.edu.ar/handle/10915/146934 |
| Access Level: | acceso abierto |
| Palabra clave: | Ciencias Informáticas Big Data Machine Learning Classification Models Apache Spark Spark ML Wilcoxon Test Student’s T Test Aprendizaje automático Modelos de clasificación Test de Wilcoxon Test T-Student |
| Sumario: | Feature selection (FS) techniques generally require repeatedly training and evaluating models to assess the importance of each feature for a particular task. However, due to the increasing size of currently available databases, distributed processing has become a necessity for many tasks. In this context, the Apache Spark ML library is one of the most widely used libraries for performing classification and other tasks with large datasets. Therefore, knowing both the predictive performance and efficiency of its main algorithms before applying a FS technique is crucial to planning computations and saving time. In this work, a comparative study of four Spark ML classification algorithms is carried out, statistically measuring execution times and predictive power based on the number of attributes from a colon cancer database. Results were statistically analyzed, showing that, although Random Forest and Naive Bayes are the algorithms with the shortest execution times, Support Vector Machine obtains models with the best predictive power. The study of the performance of these algorithms is interesting as they are applied in many different problems, such as classification of pathologies from epigenomic data, image classification, prediction of computer attacks in network security problems, among others. |
|---|