UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation

Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun

Abstract

LiDAR provides accurate geometric measurements of the 3D world, but high-density sensors are costly, and low-beam alternatives produce sparse point clouds. We present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, generation, and manipulation. By utilizing a compact, discrete representation that encodes geometric structures into a codebook, UltraLiDAR can densify sparse scans and generate realistic new environments. Our experiments demonstrate that UltraLiDAR significantly improves downstream perception performance and produces results that human participants prefer over 98% of the time compared to existing methods.

1. Introduction

In the domain of autonomous driving, LiDAR is the primary modality for precise 3D perception. However, a significant gap exists between affordable low-beam sensors and the high-density data required for robust long-range detection. Sparse point clouds often lack the structural detail necessary for identifying small objects or distant obstacles.

UltraLiDAR addresses these challenges by treating 3D scenes as compositions of discrete geometric building blocks. By learning a "vocabulary" of 3D shapes, the framework enables high-fidelity LiDAR completion (sparse-to-dense), unconditional scene generation, and intuitive scene manipulation.

2. The UltraLiDAR Framework

The core innovation of UltraLiDAR is its discrete, codebook-based representation. Rather than operating on raw, unstructured point clouds, the system employs a three-stage pipeline.

2.1 Discrete Representation via VQ-VAE

The model first voxelizes the 3D scene into a Bird’s-Eye View (BEV) grid. Using a Vector Quantized Variational Autoencoder (VQ-VAE), these grids are mapped into a discrete codebook. This process effectively compresses complex 3D geometry into a grid of "visual words"—discrete tokens representing specific geometric primitives like curb segments, vehicle corners, or road surfaces. This discretization makes the model inherently robust to noise, as it can only reconstruct scenes using valid entries from its learned vocabulary.

2.2 Perceptual Loss and Object Awareness

To ensure the compressed representation preserves features critical for autonomous navigation, UltraLiDAR incorporates a perceptual loss. By utilizing a pre-trained 3D object detector, the model is penalized if the reconstructed point cloud loses the geometric characteristics necessary for object recognition. This ensures that the "building blocks" prioritize task-relevant structures over raw pixel-wise accuracy.

2.3 Generative Transformer and Free Space Suppression

For scene generation, UltraLiDAR employs a bi-directional Transformer (similar to MaskGIT). The model predicts the most likely code for each cell in the BEV grid. A key challenge in LiDAR generation is the prevalence of empty space; to prevent the model from over-generating "blank" tokens, the authors introduced Free Space Suppression Sampling. This technique encourages the emergence of structured objects during the early stages of the iterative generation process.

3. Experimental Evaluation

The framework was evaluated across several benchmark datasets, including PandaSet, KITTI-360, and SemanticKITTI.

3.1 LiDAR Completion and Perception

UltraLiDAR was tested on its ability to densify sparse 64-beam data into 512-beam equivalent scans. When a standard 3D detector (PointPillar) was applied to the densified output, there was a significant jump in Average Precision (AP) compared to detections on raw sparse data. This suggests that UltraLiDAR successfully recovers missing geometric information that is vital for downstream perception tasks.

3.2 Realistic Scene Generation

In unconditional generation tasks, UltraLiDAR outperformed previous state-of-the-art models like LiDARGen. In human A/B testing, participants preferred UltraLiDAR’s generated scenes 98.5% of the time. Furthermore, in a "Turing Test" scenario, participants mistook generated scans for real-world data 32% of the time, approaching the 50% threshold of indistinguishability.

3.3 Scene Manipulation

The discrete nature of the representation allows for "Actor Insertion." Researchers can extract the codes representing an object (e.g., a car) from one scene and insert them into another. The decoder then generates a realistic LiDAR scan of the modified scene, correctly handling the beam patterns and geometric integration.

4. Conclusion and Future Work

UltraLiDAR demonstrates that discrete latent representations are highly effective for modeling 3D LiDAR data. By learning a robust geometric prior, the framework enables high-quality completion and generation that benefits both simulation and real-world perception.

Future research directions include incorporating Range View information to better model sensor-specific physics and occlusions, as well as extending the framework to temporal sequences to ensure consistency across video-like LiDAR streams. UltraLiDAR paves the way for scalable, data-driven simulation environments for the next generation of autonomous vehicles.

Summary updated

Abstract

LiDAR provides accurate geometric measurements of the 3D world. Unfortunately, dense LiDARs are very expensive and the point clouds captured by low-beam LiDAR are often sparse. To address these issues, we present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. The crux of UltraLiDAR is a compact, discrete representation that encodes the point cloud's geometric structure, is robust to noise, and is easy to manipulate. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds as if they were captured by a real high-density LiDAR, drastically reducing the cost. Furthermore, by learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving. We evaluate the effectiveness of UltraLiDAR on sparse-to-dense LiDAR completion and LiDAR generation. Experiments show that densifying real-world point clouds with our approach can significantly improve the performance of downstream perception systems. Compared to prior art on LiDAR generation, our approach generates much more realistic point clouds. According to A/B test, over 98.5\% of the time human participants prefer our results over those of previous methods.
BibTeX
@inproceedings{xiong2023ultralidar,
  author = {Yuwen Xiong and Wei-Chiu Ma and Jingkang Wang and Raquel Urtasun},
  title = {{UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=IiPIuvRbOJ},
}

Loading related content...

Figure