A Multi-Start Simulated Annealing Strategy for Data Lake Organization Problem
Data Lake, Organização de dado, Simulated Annealing
The Data Lake Organization Problem comprises optimized data navigation structures generation to reduce the user's time exploring all available data. The goal is to find a data organization that maximizes the expected probability of table discovery during user navigation. For this problem, we propose a simulated annealing metaheuristic and compare it with the Organize literature solution on benchmark instances. The instances are Socrata Open Data Lake samples with varying topics and open data from government entities worldwide. To validate our proposal, we performed a statistical analysis using a non-parametric test, which confirmed the dominance of our proposition over the state-of-the-art. Our proposal was more efficient and increased the expected probability of table discovery up to 15%.