Learning Compact Representations for LiDAR Completion and Generation

Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun

Abstract

LiDAR provides accurate geometric measurements of the 3D world. Unfortunately, dense LiDARs are very expensive and the point clouds captured by low-beam LiDAR are often sparse. To address these issues, we present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. The crux of UltraLiDAR is a compact, discrete representation that encodes the point cloud's geometric structure, is robust to noise, and is easy to manipulate. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds as if they were captured by a real high-density LiDAR, drastically reducing the cost. Furthermore, by learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving. We evaluate the effectiveness of UltraLiDAR on sparse-to-dense LiDAR completion and LiDAR generation. Experiments show that densifying real-world point clouds with our approach can significantly improve the performance of downstream perception systems. Compared to prior art on LiDAR generation, our approach generates much more realistic point clouds. According to A/B test, over 98.5% of the time human participants prefer our results over those of previous methods.

Introduction

LiDAR sensors serve as the primary "eyes" for autonomous vehicles, offering precise geometric data essential for safe navigation. However, current LiDAR technology faces two significant challenges: sparsity and cost. Point clouds naturally thin out at long ranges, and lower-cost sensors (e.g., 32-beam) provide significantly less detail than their high-end counterparts (e.g., 128-beam). This sparsity hinders object detection for distant or small actors. Furthermore, high-density sensors are prohibitively expensive for mass-market deployment, and creating realistic synthetic LiDAR data for simulation remains a complex, manual task.

The UltraLiDAR Framework

UltraLiDAR addresses these challenges by learning a compact, discrete representation of 3D scenes. The core innovation is the use of a Vector Quantized Variational Autoencoder (VQ-VAE) that operates on Bird’s-Eye-View (BEV) projections of voxelized point clouds.

UltraLiDAR architecture for (a) sparse-to-dense completion and (b) point cloud generation.

Discrete Representation via VQ-VAE

Rather than processing raw 3D points directly, which is computationally intensive, UltraLiDAR voxelizes the point cloud and projects it into a 2D BEV image. This allows the model to leverage efficient 2D convolutions to map the scene into a discrete "codebook" of geometric features. This discrete representation is inherently robust to sensor noise and distribution shifts, facilitating better generalization across different environments.

Sparse-to-Dense Completion

To "densify" a sparse scan (e.g., 64-beam) into a high-density one (e.g., 512-beam), the model learns to map sparse input features to the same discrete codes used by a dense decoder. This process acts as a structural prior, ensuring that the completed scene adheres to realistic physical constraints, such as occlusions and typical road layouts.

Generative Modeling

For unconditional or conditional scene generation, UltraLiDAR employs a bi-directional Transformer (similar to MaskGIT). The Transformer predicts code indices on a spatial "canvas," iteratively filling in the scene from scratch or based on partial inputs. A key technique, "Free Space Suppression," is used during sampling to prevent the model from defaulting to empty space, forcing it to prioritize the generation of meaningful structures like vehicles and buildings.

Applications and Results

The versatility of the UltraLiDAR framework enables a wide range of tasks, from densifying real-world data to manipulating existing scenes.

Qualitative results of UltraLiDAR across completion, manipulation, and generation tasks.

Enhancing Downstream Perception

Experiments on datasets like PandaSet and KITTI-360 demonstrate that densifying sparse point clouds with UltraLiDAR significantly boosts the performance of 3D object detectors. In zero-shot cross-dataset evaluations (training on PandaSet and testing on KITTI), the densified data improved detection Average Precision (AP) by 10-15% compared to raw sparse inputs.

High-Fidelity Generation

In human A/B studies, participants preferred UltraLiDAR’s generated scenes over previous state-of-the-art methods, such as LiDARGen, more than 98.5% of the time. Furthermore, human testers mistook generated scans for real data in 32% of trials, approaching the 50% threshold of a perfect "Turing test."

Scene Manipulation

The spatially aligned nature of the discrete codebook allows for intuitive "copy-paste" manipulation. Users can extract the codes for an actor (e.g., a car) and insert them into a new location or remove them entirely, enabling the creation of complex counterfactual safety scenarios for testing autonomous systems.

Conclusion and Future Work

UltraLiDAR demonstrates that learning a discrete geometric prior is a powerful approach for handling the inherent limitations of LiDAR data. While the current voxelization process may lose some fine-grained detail, the benefits in robustness and controllability are substantial. Future work includes automating the mapping of discrete codes to semantic labels and integrating range-view information to further enhance the fidelity of generated beam patterns.

Summary updated

Abstract

LiDAR provides accurate geometric measurements of the 3D world. Unfortunately, dense LiDARs are very expensive and the point clouds captured by low-beam LiDAR are often sparse. To address these issues, we present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. The crux of UltraLiDAR is a compact, discrete representation that encodes the point cloud's geometric structure, is robust to noise, and is easy to manipulate. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds as if they were captured by a real high-density LiDAR, drastically reducing the cost. Furthermore, by learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving. We evaluate the effectiveness of UltraLiDAR on sparse-to-dense LiDAR completion and LiDAR generation. Experiments show that densifying real-world point clouds with our approach can significantly improve the performance of downstream perception systems. Compared to prior art on LiDAR generation, our approach generates much more realistic point clouds. According to A/B test, over 98.5% of the time human participants prefer our results over those of previous methods. Please refer to project page https://waabi.ai/research/ultralidar/ for more information.
BibTeX
@inproceedings{xiong2023learning,
  author = {Yuwen Xiong and Wei-Chiu Ma and Jingkang Wang and Raquel Urtasun},
  title = {{Learning Compact Representations for LiDAR Completion and Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=IiPIuvRbOJ},
}
Figure