Abstract
Building models capable of generating structured output is a key challenge for AI and robotics. While generative models have been explored on many types of data, little work has been done on synthesizing LiDAR scans, which play a key role in robot mapping and localization. In this work, we show that one can adapt deep generative models for this task by unravelling LiDAR scans into a 2D point map. Our approach can generate high-quality samples while simultaneously learning a meaningful latent representation of the data. We demonstrate significant improvements against state-of-the-art point cloud generation methods. Furthermore, we propose a novel data representation that augments the 2D signal with absolute positional information, showing that this helps robustness to noisy and imputed input.
Introduction
While generative modeling has flourished in the realm of RGB images, LiDAR data—the "eyes" of most autonomous vehicles—has remained a challenge. LiDAR scans are typically represented as unordered sets of 3D coordinates (point clouds). Processing these directly is difficult: voxel-based methods are computationally expensive and waste memory on empty space, while point-based methods often ignore the inherent spatial structure provided by the laser's scanning pattern. This paper introduces a method to apply deep generative models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), to raw LiDAR data by "unrolling" 3D point clouds into 2D spherical maps.
Methodology: Unrolling the Scan
The core innovation of this work is the data representation. Instead of treating the scan as a random cloud of points, the authors leverage the physical nature of the sensor. A LiDAR sensor rotates, firing lasers at specific elevation and azimuth angles. By mapping these angles to the rows and columns of a 2D grid, they transform a 3D scan into a $H \times W \times 3$ image-like tensor.
Key technical components include:
- 2D Projection: Points are clustered by elevation and sorted by azimuth. Each "pixel" in the resulting 2D map stores the $(x, y, z)$ coordinates of the point.
- Generative Architectures: The researchers utilize VAEs for reconstruction tasks and Relativistic Average GANs (RaGAN) for high-fidelity unconditional sampling. Both models use deep convolutional architectures (DCGAN backbone), treating the LiDAR map like a multi-channel image to exploit local correlations.
Robustness and the Cartesian Advantage
A major finding of the paper is that the choice of coordinate system significantly impacts model robustness. The authors compared representing the signal as Cartesian $(x, y, z)$ coordinates versus Polar (range and elevation) coordinates. They discovered that the Cartesian representation is significantly more robust to noise. Providing absolute positional information acts as a strong inductive bias, helping the network "fill in the blanks" when data is missing or heavily distorted.
The VAE was able to reconstruct clean scans from input so noisy it appeared uninformative to the human eye, despite never being explicitly trained on "noisy-to-clean" pairs.
Key Results
The proposed model was evaluated using the KITTI dataset, a standard benchmark for autonomous driving.
- Reconstruction Performance: In clean data reconstruction, the model achieved an Earth Mover’s Distance (EMD) of 127.0, vastly outperforming baselines like AtlasNet (1571.2).
- Robustness: When 15% of the data was removed, the Cartesian model maintained high performance, while the Polar model's error increased significantly.
- Qualitative GAN Samples: The GAN successfully generated realistic road scenes, capturing complex features like "LiDAR shadows" (empty areas behind objects) and varying point densities based on distance from the sensor.
Conclusion
By treating LiDAR scans as structured 2D grids, this approach enables the use of powerful convolutional generative models. The transition from unordered point clouds to structured maps not only improves efficiency but also allows for remarkable robustness in data recovery. Future work may focus on designing loss functions that better account for non-uniform point density and extending the model to handle temporal consistency in dynamic environments.
Summary updated