A cost-based storage format selector for materialized results in big data frameworks

Modern big data frameworks (such as Hadoop and Spark) allow multiple users to do large-scale analysis simultaneously, by deploying data-intensive workflows (DIWs). These DIWs of different users share many common tasks (i.e, 50–80%), which can be materialized and reused in future executions. Material...

ver descrição completa

Detalhes bibliográficos
Autores: Munir, Rana Faisal|||0000-0003-3949-0353, Abelló Gamazo, Alberto|||0000-0002-3223-2186, Romero Moral, Óscar|||0000-0001-6350-8328, Thiele, Maik, Lehner, Wolfgang
Formato: artículo
Fecha de publicación:2019
País:España
Recursos:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/134838
Acesso em linha:https://hdl.handle.net/2117/134838
https://dx.doi.org/10.1007/s10619-019-07271-0
Access Level:acceso abierto
Palavra-chave:File organization (Computer science)
Big data
Data-intensive workflows
Materialized results
Storage format
HDFS
Cost model
Fitxers informàtics -- Oganització
Macrodades
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació::Emmagatzematge i recuperació de la informació
id ES_24f72c9c4cc4aeb7eb0fa85ed4c0d409
oai_identifier_str oai:upcommons.upc.edu:2117/134838
network_acronym_str ES
network_name_str España
repository_id_str
spelling A cost-based storage format selector for materialized results in big data frameworksMunir, Rana Faisal|||0000-0003-3949-0353Abelló Gamazo, Alberto|||0000-0002-3223-2186Romero Moral, Óscar|||0000-0001-6350-8328Thiele, MaikLehner, WolfgangFile organization (Computer science)Big dataData-intensive workflowsMaterialized resultsStorage formatHDFSCost modelFitxers informàtics -- OganitzacióMacrodadesÀrees temàtiques de la UPC::Informàtica::Sistemes d'informació::Emmagatzematge i recuperació de la informacióModern big data frameworks (such as Hadoop and Spark) allow multiple users to do large-scale analysis simultaneously, by deploying data-intensive workflows (DIWs). These DIWs of different users share many common tasks (i.e, 50–80%), which can be materialized and reused in future executions. Materializing the output of such common tasks improves the overall processing time of DIWs and also saves computational resources. Current solutions for materialization store data on Distributed File Systems by using a fixed storage format. However, a fixed choice is not the optimal one for every situation. Specifically, different layouts (i.e., horizontal, vertical or hybrid) have a huge impact on execution, according to the access patterns of the subsequent operations. In this paper, we present a cost-based approach that helps deciding the most appropriate storage format in every situation. A generic cost-based framework that selects the best format by considering the three main layouts is presented. Then, we use our framework to instantiate cost models for specific Hadoop storage formats (namely SequenceFile, Avro and Parquet), and test it with two standard benchmark suits. Our solution gives on average 1.33× speedup over fixed SequenceFile, 1.11× speedup over fixed Avro, 1.32× speedup over fixed Parquet, and overall, it provides 1.25× speedup.Peer Reviewed20192019-05-0820192019-06-20journal articlehttp://purl.org/coar/resource_type/c_6501AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/134838https://dx.doi.org/10.1007/s10619-019-07271-0reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/1348382026-05-27T15:37:01Z
dc.title.none.fl_str_mv A cost-based storage format selector for materialized results in big data frameworks
title A cost-based storage format selector for materialized results in big data frameworks
spellingShingle A cost-based storage format selector for materialized results in big data frameworks
Munir, Rana Faisal|||0000-0003-3949-0353
File organization (Computer science)
Big data
Data-intensive workflows
Materialized results
Storage format
HDFS
Cost model
Fitxers informàtics -- Oganització
Macrodades
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació::Emmagatzematge i recuperació de la informació
title_short A cost-based storage format selector for materialized results in big data frameworks
title_full A cost-based storage format selector for materialized results in big data frameworks
title_fullStr A cost-based storage format selector for materialized results in big data frameworks
title_full_unstemmed A cost-based storage format selector for materialized results in big data frameworks
title_sort A cost-based storage format selector for materialized results in big data frameworks
dc.creator.none.fl_str_mv Munir, Rana Faisal|||0000-0003-3949-0353
Abelló Gamazo, Alberto|||0000-0002-3223-2186
Romero Moral, Óscar|||0000-0001-6350-8328
Thiele, Maik
Lehner, Wolfgang
author Munir, Rana Faisal|||0000-0003-3949-0353
author_facet Munir, Rana Faisal|||0000-0003-3949-0353
Abelló Gamazo, Alberto|||0000-0002-3223-2186
Romero Moral, Óscar|||0000-0001-6350-8328
Thiele, Maik
Lehner, Wolfgang
author_role author
author2 Abelló Gamazo, Alberto|||0000-0002-3223-2186
Romero Moral, Óscar|||0000-0001-6350-8328
Thiele, Maik
Lehner, Wolfgang
author2_role author
author
author
author
dc.subject.none.fl_str_mv File organization (Computer science)
Big data
Data-intensive workflows
Materialized results
Storage format
HDFS
Cost model
Fitxers informàtics -- Oganització
Macrodades
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació::Emmagatzematge i recuperació de la informació
topic File organization (Computer science)
Big data
Data-intensive workflows
Materialized results
Storage format
HDFS
Cost model
Fitxers informàtics -- Oganització
Macrodades
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació::Emmagatzematge i recuperació de la informació
description Modern big data frameworks (such as Hadoop and Spark) allow multiple users to do large-scale analysis simultaneously, by deploying data-intensive workflows (DIWs). These DIWs of different users share many common tasks (i.e, 50–80%), which can be materialized and reused in future executions. Materializing the output of such common tasks improves the overall processing time of DIWs and also saves computational resources. Current solutions for materialization store data on Distributed File Systems by using a fixed storage format. However, a fixed choice is not the optimal one for every situation. Specifically, different layouts (i.e., horizontal, vertical or hybrid) have a huge impact on execution, according to the access patterns of the subsequent operations. In this paper, we present a cost-based approach that helps deciding the most appropriate storage format in every situation. A generic cost-based framework that selects the best format by considering the three main layouts is presented. Then, we use our framework to instantiate cost models for specific Hadoop storage formats (namely SequenceFile, Avro and Parquet), and test it with two standard benchmark suits. Our solution gives on average 1.33× speedup over fixed SequenceFile, 1.11× speedup over fixed Avro, 1.32× speedup over fixed Parquet, and overall, it provides 1.25× speedup.
publishDate 2019
dc.date.none.fl_str_mv 2019
2019-05-08
2019
2019-06-20
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/134838
https://dx.doi.org/10.1007/s10619-019-07271-0
url https://hdl.handle.net/2117/134838
https://dx.doi.org/10.1007/s10619-019-07271-0
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869404730233454592
score 15.301603

Registos relacionados