TL;DR
This paper provides a comprehensive survey of data lake concepts, architectures, and systems, aiming to clarify ambiguities and offer a structured overview for researchers and practitioners. It introduces a novel three-tier functional architecture and a classification scheme to systematically analyze existing data lake solutions, highlighting key challenges and future research directions.
What Problem Does This Paper Address?
The rapid emergence of "data lakes" has led to a proliferation of systems and definitions, creating ambiguity and a lack of a coherent understanding of their functions and available technologies. Traditional "schema-on-write" approaches like data warehouses struggle with the volume, variety, and velocity of big data. While NoSQL systems offer "schema-on-read" capabilities, a unified framework for managing and analyzing diverse data in its raw format is still missing. This survey aims to bridge this gap by providing a structured overview of the data lake landscape, from fundamental concepts to specific system implementations and future challenges.
Methods
The authors propose a three-tier functional architecture for data lakes: Ingestion, Maintenance, and Exploration. This architecture serves as the basis for a three-level classification of existing data lake solutions: by tier (when the function is needed), by function (what the function is), and by method (how the function is achieved). The survey systematically reviews and categorizes existing research and systems based on these functions, providing a detailed comparison. They also discuss various architectural philosophies like "pond" and "zone" architectures and explore different storage strategies (file-based, single data store, polystores).
Datasets
Not discussed in the paper. The paper focuses on surveying existing systems and approaches rather than introducing new datasets or benchmarks.
Key Insights and Ideas
- Data Lake Ambiguity: The term "data lake" is often used loosely, leading to confusion about its precise definition and capabilities. This survey aims to bring clarity.
- Functional Architecture is Key: A structured, function-oriented architecture is crucial for understanding and building effective data lakes. The proposed three-tier model (Ingestion, Maintenance, Exploration) provides a useful framework.
- Beyond Storage: While storage is a foundational aspect, the true value of a data lake lies in its functions for data management, processing, and analytics.
- Metadata is Paramount: The "data swamp" problem highlights the critical need for robust metadata management and data governance to make data lakes usable and valuable.
- Evolving Landscape: The data lake concept has evolved significantly since its inception, moving from simple raw data repositories to more sophisticated systems incorporating AI, crowdsourcing, and advanced analytics.
- Machine Learning Integration is the Future: A major future direction is the integration of machine learning into data lakes, addressing challenges like training data heterogeneity and enabling "in-lake" machine learning.
Key Results
The paper provides a comprehensive classification of data lake functions and systems, categorizing over 150 systems based on their contributions to ingestion, maintenance, and exploration. It offers a detailed comparison of approaches for specific functions like metadata extraction, dataset organization, related dataset discovery, data cleaning, schema evolution, data provenance, data integration, and heterogeneous data querying. The survey highlights the strengths and weaknesses of various techniques and systems, offering a valuable resource for researchers and practitioners.
Limitations and Future Work
The authors acknowledge that their survey focuses on systems that explicitly claim to be data lakes or provide partial functions, and it is beyond the scope of one article to list all possible research topics and solutions. They also note that some research topics, like data integration and data discovery, have dedicated surveys, but this paper focuses on their application within the data lake context.
Future work directions highlighted include:
- ML-Aware Data Lakes: Developing data lakes that are optimized for machine learning workflows, including better handling of heterogeneous training data and enabling "in-lake" ML training and inference.
- Advanced Analytics and Transaction Management: Integrating traditional database and data warehouse functions like transaction management and query optimization into data lakes, potentially through Lakehouse architectures.
- Cloud-Native Data Lakes: Further research into building and optimizing data lakes on cloud platforms.
- Data Provenance and Security: Continued focus on robust data provenance and security mechanisms.
Summary updated