DOD-ETL: distributed on-demand ETL for near real-time business intelligence

The competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no lo...

Descripción completa

Detalles Bibliográficos
Autores: Gustavo V. Machado, Ítalo Cunha, Adriano Cesar Machado Pereira, Leonardo B. Oliveira
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2019
País:Brasil
Institución:Universidade Federal de Minas Gerais (UFMG)
Repositorio:Repositório Institucional da UFMG
Idioma:inglés
OAI Identifier:oai:repositorio.ufmg.br:1843/68565
Acceso en línea:https://doi.org/10.1186/s13174-019-0121-z
http://hdl.handle.net/1843/68565
http://orcid.org/0000-0002-6433-171X
Access Level:acceso abierto
Palabra clave:Near real-time ETL
Business intelligence
Big data
Ciência da Computação
id BR_c8a6fe24802e90ee10bfcd821b3f9ec2
oai_identifier_str oai:repositorio.ufmg.br:1843/68565
network_acronym_str BR
network_name_str Brasil
repository_id_str
spelling DOD-ETL: distributed on-demand ETL for near real-time business intelligenceNear real-time ETLBusiness intelligenceBig dataCiência da ComputaçãoBig dataBusiness intelligenceThe competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no longer the only goal. The time frame to extract information from data determines its usefulness. This work proposes DOD-ETL, a tool that addresses, in an innovative manner, the main bottleneck in Business Intelligence solutions, the Extract Transform Load process (ETL), providing it in near real-time. DOD-ETL achieves this by combining an on-demand data stream pipeline with a distributed, parallel and technology-independent architecture with in-memory caching and efficient data partitioning. We compared DOD-ETL with other Stream Processing frameworks used to perform near real-time ETL and found DOD-ETL executes workloads up to 10 times faster. We have deployed it in a large steelworks as a replacement for its previous ETL solution, enabling near real-time reports previously unavailable.Universidade Federal de Minas GeraisBrasilICEX - INSTITUTO DE CIÊNCIAS EXATASICX - DEPARTAMENTO DE CIÊNCIA DA COMPUTAÇÃOUFMG2024-05-22T22:00:23Z2024-05-22T22:00:23Z2019-11-20info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articlepdfapplication/pdfhttps://doi.org/10.1186/s13174-019-0121-z1867-4828http://hdl.handle.net/1843/68565http://orcid.org/0000-0002-6433-171XengJournal of Internet Services and Applicationsinfo:eu-repo/semantics/openAccessreponame:Repositório Institucional da UFMGinstname:Universidade Federal de Minas Gerais (UFMG)instacron:UFMGGustavo V. MachadoÍtalo CunhaAdriano Cesar Machado PereiraLeonardo B. Oliveira2024-05-22T22:00:24Zoai:repositorio.ufmg.br:1843/68565Repositório InstitucionalPUBhttps://repositorio.ufmg.br/oairepositorio@ufmg.bropendoar:2024-05-22T22:00:24Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG)false
dc.title.none.fl_str_mv DOD-ETL: distributed on-demand ETL for near real-time business intelligence
title DOD-ETL: distributed on-demand ETL for near real-time business intelligence
spellingShingle DOD-ETL: distributed on-demand ETL for near real-time business intelligence
Gustavo V. Machado
Near real-time ETL
Business intelligence
Big data
Ciência da Computação
Big data
Business intelligence
title_short DOD-ETL: distributed on-demand ETL for near real-time business intelligence
title_full DOD-ETL: distributed on-demand ETL for near real-time business intelligence
title_fullStr DOD-ETL: distributed on-demand ETL for near real-time business intelligence
title_full_unstemmed DOD-ETL: distributed on-demand ETL for near real-time business intelligence
title_sort DOD-ETL: distributed on-demand ETL for near real-time business intelligence
dc.creator.none.fl_str_mv Gustavo V. Machado
Ítalo Cunha
Adriano Cesar Machado Pereira
Leonardo B. Oliveira
author Gustavo V. Machado
author_facet Gustavo V. Machado
Ítalo Cunha
Adriano Cesar Machado Pereira
Leonardo B. Oliveira
author_role author
author2 Ítalo Cunha
Adriano Cesar Machado Pereira
Leonardo B. Oliveira
author2_role author
author
author
dc.subject.por.fl_str_mv Near real-time ETL
Business intelligence
Big data
Ciência da Computação
Big data
Business intelligence
topic Near real-time ETL
Business intelligence
Big data
Ciência da Computação
Big data
Business intelligence
description The competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no longer the only goal. The time frame to extract information from data determines its usefulness. This work proposes DOD-ETL, a tool that addresses, in an innovative manner, the main bottleneck in Business Intelligence solutions, the Extract Transform Load process (ETL), providing it in near real-time. DOD-ETL achieves this by combining an on-demand data stream pipeline with a distributed, parallel and technology-independent architecture with in-memory caching and efficient data partitioning. We compared DOD-ETL with other Stream Processing frameworks used to perform near real-time ETL and found DOD-ETL executes workloads up to 10 times faster. We have deployed it in a large steelworks as a replacement for its previous ETL solution, enabling near real-time reports previously unavailable.
publishDate 2019
dc.date.none.fl_str_mv 2019-11-20
2024-05-22T22:00:23Z
2024-05-22T22:00:23Z
dc.type.status.fl_str_mv info:eu-repo/semantics/publishedVersion
dc.type.driver.fl_str_mv info:eu-repo/semantics/article
format article
status_str publishedVersion
dc.identifier.uri.fl_str_mv https://doi.org/10.1186/s13174-019-0121-z
1867-4828
http://hdl.handle.net/1843/68565
http://orcid.org/0000-0002-6433-171X
url https://doi.org/10.1186/s13174-019-0121-z
http://hdl.handle.net/1843/68565
http://orcid.org/0000-0002-6433-171X
identifier_str_mv 1867-4828
dc.language.iso.fl_str_mv eng
language eng
dc.relation.none.fl_str_mv Journal of Internet Services and Applications
dc.rights.driver.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv pdf
application/pdf
dc.publisher.none.fl_str_mv Universidade Federal de Minas Gerais
Brasil
ICEX - INSTITUTO DE CIÊNCIAS EXATAS
ICX - DEPARTAMENTO DE CIÊNCIA DA COMPUTAÇÃO
UFMG
publisher.none.fl_str_mv Universidade Federal de Minas Gerais
Brasil
ICEX - INSTITUTO DE CIÊNCIAS EXATAS
ICX - DEPARTAMENTO DE CIÊNCIA DA COMPUTAÇÃO
UFMG
dc.source.none.fl_str_mv reponame:Repositório Institucional da UFMG
instname:Universidade Federal de Minas Gerais (UFMG)
instacron:UFMG
instname_str Universidade Federal de Minas Gerais (UFMG)
instacron_str UFMG
institution UFMG
reponame_str Repositório Institucional da UFMG
collection Repositório Institucional da UFMG
repository.name.fl_str_mv Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG)
repository.mail.fl_str_mv repositorio@ufmg.br
_version_ 1853663266319892480
score 15,300719

Similares en LA Referencia