Abstract
PillarSegNet is a novel end-to-end deep learning architecture designed to transform sparse 3D LiDAR point clouds into dense, top-view 2D semantic grid maps. By learning features directly from vertical columns of points ("pillars") rather than relying on hand-crafted grid features, it achieves a significant performance boost over previous state-of-the-art methods, particularly in identifying small objects like pedestrians and bicycles.
Introduction
For autonomous vehicles to navigate safely, they require a reliable understanding of their surroundings. While LiDAR sensors provide excellent 3D geometric data, the output is inherently sparse and unstructured. Most existing LiDAR segmentation models predict classes on a point-by-point basis. However, downstream path-planning algorithms typically prefer a dense 2D top-view grid map that explicitly labels every patch of ground (e.g., road, sidewalk, parked car, free space), even if no LiDAR point directly hit that exact spot in the current sweep.
Previous attempts to generate these dense semantic grid maps relied on hand-crafted feature extraction, which often discarded crucial geometric details. PillarSegNet aims to bridge this gap by learning features directly from the raw 3D points while still outputting a dense 2D grid map.
Methods
The authors propose PillarSegNet, a pipeline that processes a single sparse LiDAR sweep through two parallel streams:
- Point Cloud Feature Encoding (The "Pillar" Stream): Instead of manually flattening the data, the 3D space is discretized into vertical columns called "pillars." The raw points inside each pillar are fed into a simplified PointNet, allowing the network to learn rich, data-driven features directly from the 3D geometry. The resulting features are then scattered back into a 2D top-view "pseudo-image."
- Occupancy Features (The Ray-Casting Stream): Because LiDAR points only indicate where surfaces are, the authors use a physical ray-casting model to calculate an "observability map." This explicitly informs the network where space is known to be empty versus where it is occluded.
The learned pillar features and the occupancy features are concatenated and fed into a modified U-Net (an encoder-decoder network). The U-Net outputs the final dense semantic grid map, predicting probabilities for 12 different classes for every grid cell.
Datasets and Training
The model is trained and evaluated on the SemanticKITTI dataset. To train the model to predict dense maps from sparse inputs, the authors generated "Dense Ground Truth" by superimposing up to 40 neighboring LiDAR scans. This forces the network to learn how to accurately "hallucinate" and fill in the unobserved gaps of a sparse single sweep.
Key Results
PillarSegNet outperformed the previous state-of-the-art grid map method (Bieder et al.) by 15.3% mIoU when evaluated on sparse ground truth and 5.7% mIoU on dense ground truth. The improvement on small classes was particularly staggering; for example, the Intersection over Union (IoU) for the person class jumped from 0.0% to 15.8%, and the two-wheel class jumped from 0.0% to 25.8%.
The entire pipeline runs in just 58 ms (approx. 17 FPS) on an Nvidia 2080 Ti GPU, making it highly suitable for real-time autonomous driving applications.
Conclusion
By utilizing end-to-end learning from 3D pillars and integrating physical occupancy models, PillarSegNet effectively addresses the "blind spots" of traditional grid-map methods. While 3D occupancy features yield the highest accuracy, the authors suggest that 2D occupancy features provide the most practical trade-off between speed and performance for real-time systems.
Summary updated