Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data

Kazuto Nakashima, Yumi Iwashita, Ryo KurazumeWACV 2023

TL;DR

This paper introduces a novel generative model for LiDAR range images that learns to simulate realistic ray-casting and ray-dropping effects. This allows for more effective domain adaptation in LiDAR perception tasks, particularly for semantic segmentation, by bridging the gap between simulated and real-world data.

What Problem Does This Paper Address?

Autonomous robots rely heavily on LiDAR sensors for 3D perception, but deploying models trained on simulated data to real-world scenarios is challenging due to a "domain gap." This gap arises from inconsistencies in how LiDAR sensors capture data, specifically:

  • Angular Resolution: Different LiDARs have varying angular resolutions, leading to spatial biases in training data that don't generalize well.
  • Ray-Dropping: LiDAR sensors can miss points due to factors like low reflectivity, specular reflections, or absorption. This "ray-dropping" artifact is difficult to accurately replicate in simulators, making simulated data less realistic.

Existing methods for domain adaptation often struggle with these issues, either by being limited by training configurations or by using simplified noise models that don't capture the nuances of real-world ray-dropping.

Methods

The authors propose a generative adversarial network (GAN) that learns to create realistic LiDAR range images by incorporating two key ideas:

  • Implicit Neural Representation: Instead of directly generating a fixed-resolution image, their generator learns a continuous, resolution-agnostic representation of the scene. This allows them to query the scene at arbitrary resolutions, making the model adaptable to different sensor specifications.
  • Differentiable Ray-Dropping: They model the ray-dropping phenomenon as a stochastic process with scene-specific probabilities. By making this process differentiable, they can train the GAN end-to-end to learn both the underlying complete scene and the probability of missing points. This is achieved by combining a complete range image prediction with a ray-drop probability map, and then sampling a final, potentially incomplete, range image.

The model is trained using a GAN framework, where the generator learns to produce realistic LiDAR data, and a discriminator tries to distinguish between real and generated data. They also introduce a method for "GAN inversion" to reconstruct and compensate for partially observed LiDAR data.

Datasets

  • KITTI Raw dataset: Used for training and evaluating the generative model's fidelity and diversity. The dataset provides 22 trajectories of scans from a Velodyne HDL-64E LiDAR.
  • GTA-LiDAR dataset: A large synthetic dataset (120k range images) with pixel-wise annotations for car and pedestrian classes, used for training the semantic segmentation model.
  • KITTI-frontal dataset: A subset of the KITTI dataset (10k real range images) used for evaluating the Sim2Real performance of the semantic segmentation model.

Key Insights and Ideas

  • Implicit Representation is Key for Adaptability: Learning a continuous, resolution-agnostic scene representation is crucial for handling varying LiDAR sensor specifications and for tasks like upsampling.
  • Realistic Ray-Dropping is Essential for Sim2Real: Simply ignoring ray-dropping or using simplistic noise models is insufficient. Accurately simulating instance-level ray-drops, influenced by scene geometry, significantly improves the performance of models trained on simulated data.
  • GANs as Scene Priors: The trained generative model acts as a powerful "scene prior," enabling not only realistic data generation but also applications like data restoration and upsampling.

Key Results

  • Generation Fidelity and Diversity: The proposed model outperformed both point-based and image-based generative baselines on various metrics, including Fréchet Point Cloud Distance (FPD), Squared Maximum Mean Discrepancy (MMD), and Sliced Wasserstein Distance (SWD). For instance, their model achieved an FPD of 93.85 and an MMD of 3.84 on full-resolution point clouds, significantly better than baselines.
  • Data Upsampling and Restoration: The model successfully demonstrated the ability to upsample LiDAR range images to higher resolutions (2x, 4x) and restore sparse depth observations.
  • Sim2Real Semantic Segmentation: When used to generate realistic ray-drop noises for simulated data, the model achieved the best mean Intersection over Union (mIoU) of 46.3% on the KITTI-frontal dataset, outperforming other methods that used simpler noise models or domain adaptation techniques. This was particularly evident in reducing false negatives for car detection.

Limitations and Future Work

  • Limitations: While the model shows strong performance, the paper notes that the 1-WGAN baseline achieved a better MMD score, which they attribute to its direct optimization of EMD during autoencoder training. The current work focuses on specific LiDAR sensor characteristics and may require adaptation for drastically different sensor types.
  • Future Work: The authors suggest exploring domain adaptation between different LiDAR sensors and leveraging larger, more diverse datasets for training. They also envision their sensor-agnostic scene representation being useful for cross-dataset tasks.

Summary updated

BibTeX
@inproceedings{nakashima2023generative,
  author = {Kazuto Nakashima and Yumi Iwashita and Ryo Kurazume},
  title = {{Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure