Point Cloud Data Management for Analytics in a Lakehouse

Balthasar Teuscher, Martin WernerAGILE Conference 2025 · DOI

TL;DR

This paper introduces a novel approach to managing and analyzing massive point cloud datasets by leveraging the "Lakehouse" architectural pattern. By treating point clouds as raw data files within a query engine, it enables scalable analytics and visualization with an SQL and DataFrame interface, offering a simpler and more flexible alternative to traditional specialized solutions.

What Problem Does This Paper Address?

Point cloud data, generated from sources like LiDAR and 3D scanning, is becoming increasingly prevalent and voluminous. While specialized tools exist for managing and analyzing this data, they often struggle with scalability and interoperability with the broader big data analytics ecosystem. This leads to complex and time-consuming data transformation and ingestion processes, data duplication, and a general lack of seamless integration. The authors aim to bridge this gap by proposing a unified approach that makes point cloud data readily accessible for analytics at scale, inspired by the Lakehouse pattern.

Methods

The core of the proposed solution is the application of the Lakehouse architectural pattern to point cloud data. This involves:

  • NoDB Philosophy: Directly accessing raw point cloud files (like LAS/LAZ) through a query engine, eliminating the need for traditional database ingestion and transformation.
  • Columnar File Formats: Utilizing formats like Apache Parquet, which store data in a columnar fashion. This is crucial for efficient analytical queries as it allows reading only the necessary columns.
  • Query Engine Integration: Employing a query engine like DataFusion (built on Apache Arrow) to process these columnar files. This provides familiar interfaces like SQL and DataFrames.
  • Versatile Repartitioning: Introducing a novel "windowed bounding volume partitioning" algorithm. This algorithm efficiently organizes point clouds into arbitrary spatial partitions, facilitating faster data retrieval by minimizing the amount of data that needs to be processed for search predicates.
  • Importance Augmentation: For visualization purposes, points are augmented with a semantic notion of "importance" to facilitate sampling and retrieval of representative data within a given field of view, respecting GPU limitations.

Datasets

The primary dataset used for evaluation is the Actueel Hoogtebestand Nederland (AHN3), a large-scale point cloud dataset. The authors used subsets of this dataset, including one with approximately 200 million points and another with about 2 billion points, to ensure comparability with existing benchmarks. No new benchmarks were introduced; rather, the performance was compared against established methods and datasets.

Key Insights and Ideas

  • Lakehouse for Point Clouds is Viable: The paper demonstrates that the Lakehouse pattern, successful in other big data domains, can be effectively applied to point cloud data, offering a unified and scalable solution.
  • Direct File Access is Powerful: Embracing the "NoDB" philosophy by querying raw files directly significantly simplifies the data management pipeline and reduces processing overhead.
  • Columnar Formats are Key: The columnar nature of formats like Parquet is essential for enabling efficient analytical queries on point clouds, allowing for selective data reading.
  • Partitioning is Crucial for Performance: The proposed windowed bounding volume partitioning algorithm is a key enabler for fast data retrieval, allowing for flexible and efficient organization of point cloud data.
  • Generic Systems Outperform Niche Solutions: The authors suggest that leveraging generic, composable big data technologies can offer advantages over highly specialized, domain-specific solutions for point cloud management.

Key Results

  • Storage Footprint: Parquet files, especially when compressed with codecs like zstd, offer a storage footprint comparable to or only slightly larger than LAZ files, making it a practical choice.
  • Data Loading: Loading LAZ files is very fast. When converting to Parquet and applying partitioning, the time increases but remains competitive, with Parquet significantly outperforming LAZ when chunk statistics are extracted.
  • Query Performance: Parquet with hierarchical partitioning achieved query times of less than 0.5 seconds across various query types and dataset sizes (up to 2 billion points), outperforming other evaluated methods.
  • Interactive Visualization: Visualization queries using DataFusion were executed in under 300 ms, and even below 100 ms when using an R*-tree index, demonstrating the feasibility of real-time interactive visualization.

Limitations and Future Work

  • Compression Trade-offs: While compression is beneficial for storage, it can impact query performance. The authors suggest that for analytical fidelity, avoiding block compression might be preferable if storage is not a major constraint.
  • Partitioning Strategy: The performance of the windowed bounding volume partitioning algorithm is sensitive to how reading windows are defined, requiring a balance between coverage and window size.
  • Future Work: The paper implicitly suggests that further exploration of different partitioning schemes and optimization strategies within the Lakehouse framework for point clouds could yield even better performance. The authors also highlight the potential for integrating more advanced geospatial analytics and machine learning capabilities within this framework.

Summary updated

BibTeX
@inproceedings{teuscher2025point,
  author = {Balthasar Teuscher and Martin Werner},
  title = {{Point Cloud Data Management for Analytics in a Lakehouse}},
  booktitle = {AGILE Conference},
  year = {2025},
  doi = {10.5194/agile-giss-6-47-2025},
}
Figure