Automatic generation of workload profiles using unsupervised learning pipelines

The complexity of resource usage and power consumption on cloud-based applications makes the understanding of application behavior through expert examination difficult. The difficulty increases when applications are seen as “black boxes”, where only external monitoring can be retrieved. Furthermore,...

ver descrição completa

Detalhes bibliográficos
Autores: Buchaca Prats, David, Berral García, Josep Lluís|||0000-0003-3037-3580, Carrera Pérez, David|||0000-0003-4898-3424
Formato: artículo
Fecha de publicación:2017
País:España
Recursos:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/113596
Acesso em linha:https://hdl.handle.net/2117/113596
https://dx.doi.org/10.1109/TNSM.2017.2786047
Access Level:acceso abierto
Palavra-chave:Memory management (Computer science)
Hidden Markov models
Machine learning
CRBM
Deep learning
MapReduce
Measurement
Monitoring
Phase detection
Telemetry
Unsupervised learning
Workload modeling
Gestió de memòria (Informàtica)
Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
id ES_5db5317ca02b71767634f11e35ef5d4e
oai_identifier_str oai:upcommons.upc.edu:2117/113596
network_acronym_str ES
network_name_str España
repository_id_str
spelling Automatic generation of workload profiles using unsupervised learning pipelinesBuchaca Prats, DavidBerral García, Josep Lluís|||0000-0003-3037-3580Carrera Pérez, David|||0000-0003-4898-3424Memory management (Computer science)Hidden Markov modelsMachine learningCRBMDeep learningMapReduceMeasurementMonitoringPhase detectionTelemetryUnsupervised learningWorkload modelingGestió de memòria (Informàtica)Aprenentatge automàticÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàticÀrees temàtiques de la UPC::Informàtica::Arquitectura de computadorsThe complexity of resource usage and power consumption on cloud-based applications makes the understanding of application behavior through expert examination difficult. The difficulty increases when applications are seen as “black boxes”, where only external monitoring can be retrieved. Furthermore, given the different amount of scenarios and applications, automation is required. Here we examine and model application behavior by finding behavior phases. We use Conditional Restricted Boltzmann Machines (CRBM) to model time-series containing resources traces measurements like CPU, Memory and IO. CRBMs can be used to map a given given historic window of trace behaviour into a single vector. This low dimensional and time-aware vector can be passed through clustering methods, from simplistic ones like k-means to more complex ones like those based on Hidden Markov Models (HMM). We use these methods to find phases of similar behaviour in the workloads. Our experimental evaluation shows that the proposed method is able to identify different phases of resource consumption across different workloads. We show that the distinct phases contain specific resource patterns that distinguish them.Peer Reviewed20172017-12-2720182018-02-02journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/113596https://dx.doi.org/10.1109/TNSM.2017.2786047reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengEuropean Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 639595 Holistic Integration of Emerging Supercomputing TechnologiesMinisterio de Economía y Competitividad http://doi.org/10.13039/501100003329 TIN2015-65316-P COMPUTACION DE ALTAS PRESTACIONES VIIopen accesshttp://purl.org/coar/access_right/c_abf2Attribution 3.0 Spainhttp://creativecommons.org/licenses/by/3.0/es/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/1135962026-05-27T15:37:01Z
dc.title.none.fl_str_mv Automatic generation of workload profiles using unsupervised learning pipelines
title Automatic generation of workload profiles using unsupervised learning pipelines
spellingShingle Automatic generation of workload profiles using unsupervised learning pipelines
Buchaca Prats, David
Memory management (Computer science)
Hidden Markov models
Machine learning
CRBM
Deep learning
MapReduce
Measurement
Monitoring
Phase detection
Telemetry
Unsupervised learning
Workload modeling
Gestió de memòria (Informàtica)
Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
title_short Automatic generation of workload profiles using unsupervised learning pipelines
title_full Automatic generation of workload profiles using unsupervised learning pipelines
title_fullStr Automatic generation of workload profiles using unsupervised learning pipelines
title_full_unstemmed Automatic generation of workload profiles using unsupervised learning pipelines
title_sort Automatic generation of workload profiles using unsupervised learning pipelines
dc.creator.none.fl_str_mv Buchaca Prats, David
Berral García, Josep Lluís|||0000-0003-3037-3580
Carrera Pérez, David|||0000-0003-4898-3424
author Buchaca Prats, David
author_facet Buchaca Prats, David
Berral García, Josep Lluís|||0000-0003-3037-3580
Carrera Pérez, David|||0000-0003-4898-3424
author_role author
author2 Berral García, Josep Lluís|||0000-0003-3037-3580
Carrera Pérez, David|||0000-0003-4898-3424
author2_role author
author
dc.subject.none.fl_str_mv Memory management (Computer science)
Hidden Markov models
Machine learning
CRBM
Deep learning
MapReduce
Measurement
Monitoring
Phase detection
Telemetry
Unsupervised learning
Workload modeling
Gestió de memòria (Informàtica)
Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
topic Memory management (Computer science)
Hidden Markov models
Machine learning
CRBM
Deep learning
MapReduce
Measurement
Monitoring
Phase detection
Telemetry
Unsupervised learning
Workload modeling
Gestió de memòria (Informàtica)
Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors
description The complexity of resource usage and power consumption on cloud-based applications makes the understanding of application behavior through expert examination difficult. The difficulty increases when applications are seen as “black boxes”, where only external monitoring can be retrieved. Furthermore, given the different amount of scenarios and applications, automation is required. Here we examine and model application behavior by finding behavior phases. We use Conditional Restricted Boltzmann Machines (CRBM) to model time-series containing resources traces measurements like CPU, Memory and IO. CRBMs can be used to map a given given historic window of trace behaviour into a single vector. This low dimensional and time-aware vector can be passed through clustering methods, from simplistic ones like k-means to more complex ones like those based on Hidden Markov Models (HMM). We use these methods to find phases of similar behaviour in the workloads. Our experimental evaluation shows that the proposed method is able to identify different phases of resource consumption across different workloads. We show that the distinct phases contain specific resource patterns that distinguish them.
publishDate 2017
dc.date.none.fl_str_mv 2017
2017-12-27
2018
2018-02-02
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/113596
https://dx.doi.org/10.1109/TNSM.2017.2786047
url https://hdl.handle.net/2117/113596
https://dx.doi.org/10.1109/TNSM.2017.2786047
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv European Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 639595 Holistic Integration of Emerging Supercomputing Technologies
Ministerio de Economía y Competitividad http://doi.org/10.13039/501100003329 TIN2015-65316-P COMPUTACION DE ALTAS PRESTACIONES VII
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 3.0 Spain
http://creativecommons.org/licenses/by/3.0/es/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 3.0 Spain
http://creativecommons.org/licenses/by/3.0/es/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869409047625596929
score 15,300719

Registos relacionados