A staging area for in-memory computing
An in-memory staging area provides fast access to different applications. This research is based on evaluating the benefits of a distributed in-memory staging area applied to the field of Big data. With this purpose, a prototype is designed and proposed to verify the idea. Then, a working version co...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2018 |
| País: | España |
| Institución: | Universitat Oberta de Catalunya (UOC) |
| Repositorio: | O2, repositorio institucional de la UOC |
| OAI Identifier: | oai:openaccess.uoc.edu:10609/83667 |
| Acceso en línea: | http://hdl.handle.net/10609/83667 |
| Access Level: | acceso abierto |
| Palabra clave: | in-memory database big data Alluxio base de datos en memoria base de dades en memòria Big data -- TFM Dades massives -- TFM Datos masivos -- TFM |
| Sumario: | An in-memory staging area provides fast access to different applications. This research is based on evaluating the benefits of a distributed in-memory staging area applied to the field of Big data. With this purpose, a prototype is designed and proposed to verify the idea. Then, a working version comprised of the in-memory software Alluxio and the processing engine Apache Spark is deployed and evaluated. In particular, the work demonstrates the increase in performance resulting from updating the data in the in-memory staging instead of allocating space for new objects. The evaluation is conducted by running an analytic with Spark over a continuously changing dataset stored in Alluxio. The experiments reported a throughput increase of 10x when compared to storing information in a regular parallel filesystem, and an increase of 3x compared to the official deployment methodology. By updating the dataset, the Alluxio in-memory capacity stays constant at a low level compared to current deployments where its capacity decreases linearly, resulting in lower performance. |
|---|