Freyja: Efficient join discovery in data lakes

We study the problem of efficiently computing rankings of joinable attributes in data lakes. Traditional set-overlap measures produce numerous false positives in this scenario, while modern, more accurate Table Representation Learning (TRL) techniques incur prohibitive computational costs. In contra...

Full description

Bibliographic Details
Authors: Maynou Yelamos, Marc, Nadal Francesch, Sergi|||0000-0002-8565-952X, Panadero Palenzuela, Raquel, Flores Herrera, Javier de Jesús|||0000-0002-2998-9962, Romero Moral, Óscar|||0000-0001-6350-8328, Queralt Calafat, Anna|||0000-0003-2782-2955
Format: article
Publication Date:2026
Country:España
Institution:Universitat Politècnica de Catalunya (UPC)
Repository:UPCommons. Portal del coneixement obert de la UPC
Language:English
OAI Identifier:oai:upcommons.upc.edu:2117/452744
Online Access:https://hdl.handle.net/2117/452744
https://dx.doi.org/10.1109/TKDE.2026.3656786
Access Level:Open access
Keyword:Data discovery
Join discovery
Big data processing
Data lakes
Data profiling
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació
id ES_d20df7e194fc56287c582f8ba7a72776
oai_identifier_str oai:upcommons.upc.edu:2117/452744
network_acronym_str ES
network_name_str España
repository_id_str
spelling Freyja: Efficient join discovery in data lakesMaynou Yelamos, MarcNadal Francesch, Sergi|||0000-0002-8565-952XPanadero Palenzuela, RaquelFlores Herrera, Javier de Jesús|||0000-0002-2998-9962Romero Moral, Óscar|||0000-0001-6350-8328Queralt Calafat, Anna|||0000-0003-2782-2955Data discoveryJoin discoveryBig data processingData lakesData profilingÀrees temàtiques de la UPC::Informàtica::Sistemes d'informacióWe study the problem of efficiently computing rankings of joinable attributes in data lakes. Traditional set-overlap measures produce numerous false positives in this scenario, while modern, more accurate Table Representation Learning (TRL) techniques incur prohibitive computational costs. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes relying on a metric that combines multiset Jaccard and cardinality proportion. The proposed metric merges the best of both worlds by leveraging syntactic measures while achieving accuracy scores comparable to those of TRL approaches. Generating rankings of joinable pairs is highly scalable at both preparation and query time, since we train a general-purpose predictive model. Predictions are based on data profiles, succinct and efficiently computed representations of dataset characteristics. Our experiments show that our system, Freyja, matches and improves upon, the results obtained by the state-of-the-art while reducing execution costs by orders of magnitude.This work has been partly supported by the Horizon Europe Programme under GA.101135513 (CyclOps) and the Spanish Ministerio de Ciencia e Innovacion under project PID2023-152841OA-I00 / AEI/10.13039/501100011033 (TALC). Anna Queralt is a Serra Hunter Fellow.Peer Reviewed20262026-04-0120262026-02-05journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/452744https://dx.doi.org/10.1109/TKDE.2026.3656786reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengEuropean Commission http://doi.org/10.13039/501100000780 HE 101135513 Automated end-to-end data life cycle management for FAIR data integration, processing and re-useAgencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2023-152841OA-I00 HACIA UN CICLO DE VIDA AUTOMATIZADO DE DATOS CENTRADO EN LA IAopen accesshttp://purl.org/coar/access_right/c_abf2http://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4527442026-05-27T15:37:01Z
dc.title.none.fl_str_mv Freyja: Efficient join discovery in data lakes
title Freyja: Efficient join discovery in data lakes
spellingShingle Freyja: Efficient join discovery in data lakes
Maynou Yelamos, Marc
Data discovery
Join discovery
Big data processing
Data lakes
Data profiling
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació
title_short Freyja: Efficient join discovery in data lakes
title_full Freyja: Efficient join discovery in data lakes
title_fullStr Freyja: Efficient join discovery in data lakes
title_full_unstemmed Freyja: Efficient join discovery in data lakes
title_sort Freyja: Efficient join discovery in data lakes
dc.creator.none.fl_str_mv Maynou Yelamos, Marc
Nadal Francesch, Sergi|||0000-0002-8565-952X
Panadero Palenzuela, Raquel
Flores Herrera, Javier de Jesús|||0000-0002-2998-9962
Romero Moral, Óscar|||0000-0001-6350-8328
Queralt Calafat, Anna|||0000-0003-2782-2955
author Maynou Yelamos, Marc
author_facet Maynou Yelamos, Marc
Nadal Francesch, Sergi|||0000-0002-8565-952X
Panadero Palenzuela, Raquel
Flores Herrera, Javier de Jesús|||0000-0002-2998-9962
Romero Moral, Óscar|||0000-0001-6350-8328
Queralt Calafat, Anna|||0000-0003-2782-2955
author_role author
author2 Nadal Francesch, Sergi|||0000-0002-8565-952X
Panadero Palenzuela, Raquel
Flores Herrera, Javier de Jesús|||0000-0002-2998-9962
Romero Moral, Óscar|||0000-0001-6350-8328
Queralt Calafat, Anna|||0000-0003-2782-2955
author2_role author
author
author
author
author
dc.subject.none.fl_str_mv Data discovery
Join discovery
Big data processing
Data lakes
Data profiling
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació
topic Data discovery
Join discovery
Big data processing
Data lakes
Data profiling
Àrees temàtiques de la UPC::Informàtica::Sistemes d'informació
description We study the problem of efficiently computing rankings of joinable attributes in data lakes. Traditional set-overlap measures produce numerous false positives in this scenario, while modern, more accurate Table Representation Learning (TRL) techniques incur prohibitive computational costs. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes relying on a metric that combines multiset Jaccard and cardinality proportion. The proposed metric merges the best of both worlds by leveraging syntactic measures while achieving accuracy scores comparable to those of TRL approaches. Generating rankings of joinable pairs is highly scalable at both preparation and query time, since we train a general-purpose predictive model. Predictions are based on data profiles, succinct and efficiently computed representations of dataset characteristics. Our experiments show that our system, Freyja, matches and improves upon, the results obtained by the state-of-the-art while reducing execution costs by orders of magnitude.
publishDate 2026
dc.date.none.fl_str_mv 2026
2026-04-01
2026
2026-02-05
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/452744
https://dx.doi.org/10.1109/TKDE.2026.3656786
url https://hdl.handle.net/2117/452744
https://dx.doi.org/10.1109/TKDE.2026.3656786
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv European Commission http://doi.org/10.13039/501100000780 HE 101135513 Automated end-to-end data life cycle management for FAIR data integration, processing and re-use
Agencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2023-152841OA-I00 HACIA UN CICLO DE VIDA AUTOMATIZADO DE DATOS CENTRADO EN LA IA
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2

http://creativecommons.org/licenses/by/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2

http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869420316799795200
score 15,81155

Similar Items