Rethinking 3D LiDAR Point Cloud Segmentation

Shijie Li, Yun Liu, Jüergen Gall

Abstract

Many point-based semantic segmentation methods designed for indoor scenarios struggle when applied to outdoor LiDAR data. To address this, we introduce a general concept of reformulating 3D point-based operations to operate in projection space. This reformulation makes methods 300 to 400 times faster and more accurate. We further introduce UnPNet, a network that integrates these reformulated operations into a 2D encoder-decoder architecture. Evaluated on four challenging datasets, our approach achieves state-of-the-art results by unifying the benefits of point-based and image-based methods.

What Problem Does This Paper Address?

Semantic segmentation—labeling every point in a 3D scan (e.g., "car," "tree," "road")—is critical for autonomous driving. However, a major rift exists between current approaches:

  1. Point-based methods (e.g., PointNet++) capture 3D structures well but are computationally expensive ($O(N \log N)$ complexity) and struggle with the massive scale and sparsity of outdoor LiDAR data.
  2. Image-based methods project 3D data onto a 2D plane. While fast due to standard 2D convolutions, they often lose critical 3D geometric relationships.
Comparison of 3D LiDAR point clouds and their corresponding 2D projection space representations.

The authors address this efficiency-accuracy trade-off, specifically identifying that standard 3D sampling techniques like Farthest Point Sampling (FPS) are ill-suited for outdoor environments as they over-focus on distant, noisy outliers.

Methods

The core innovation is the reformulation of 3D operations to work within a 2D projection of the LiDAR scan.

Reformulated Sampling (Ray Sampling)

Instead of complex 3D sampling, the authors uniformly sample "rays" on the 2D projected grid. This reduces complexity to $O(M)$, where $M$ is the number of sampled points, making it significantly faster and more robust to outliers than traditional FPS.

Comparison between (a) Farthest Point Sampling and (b) the proposed Ray Sampling in projection space.

Reformulated Grouping

Neighbors are found using a $k \times k$ window on the 2D grid, but actual 3D coordinates are used for feature calculation. This preserves 3D geometry while leveraging 2D data structures for speed.

UnPNet Architecture

The Unprojection Network (UnPNet) integrates these operations into a 2D encoder-decoder network featuring:

  • Context Blocks: Dilated convolutions capture multi-scale information without resolution loss.
  • Edge Supervision: A specialized loss function focuses on object boundaries, which are often blurred in projections.

Datasets and Evaluation

The approach was evaluated on four major outdoor LiDAR benchmarks:

  • SemanticKITTI: The primary benchmark for large-scale outdoor segmentation.
  • SemanticPOSS: Focused on complex campus environments with many small, moving objects.
  • nuScenes: A massive-scale dataset testing robustness across different distances.
  • Pandaset: Includes data from both spinning and solid-state LiDAR sensors.

Key Results

  • Massive Speedups: Reformulating baselines like PointNet++ and PointConv made them 300 to 400 times faster.
  • Accuracy Boost: Reformulated versions saw mean Intersection-over-Union (mIoU) increases of 58% to 68%.
  • State-of-the-Art: UnPNet achieved a mIoU of 54.6 on SemanticKITTI, outperforming more complex models.
  • Robustness: The model excelled at detecting small objects like bicycles and pedestrians across all datasets.

Key Insights

  • The Best of Both Worlds: Performing 3D math on a 2D grid provides the efficiency of image processing with the spatial awareness of point-based models.
  • Sampling is the Bottleneck: Ray Sampling is better suited to the physical nature of LiDAR sensors than FPS.
  • Density Matters: Handling point sparsity through reformulated operations allows for much more effective detection of distant objects.

Limitations and Future Work

While highly efficient, projection-based methods can still suffer from occlusion where objects at different distances overlap in 2D. Future work includes applying this reformulation concept to point-voxel methods (like PV-RCNN) to bring high-end 3D detection to real-time hardware.

Summary updated

Abstract

Many point-based semantic segmentation methods have been designed for indoor scenarios, but they struggle if they are applied to point clouds that are captured by a LiDAR sensor in an outdoor environment. In order to make these methods more efficient and robust such that they can handle LiDAR data, we introduce the general concept of reformulating 3D point-based operations such that they can operate in the projection space. While we show by means of three point-based methods that the reformulated versions are between 300 and 400 times faster and achieve a higher accuracy, we furthermore demonstrate that the concept of reformulating 3D point-based operations allows to design new architectures that unify the benefits of point-based and image-based methods. As an example, we introduce a network that integrates reformulated 3D point-based operations into a 2D encoder-decoder architecture that fuses the information from different 2D scales. We evaluate the approach on four challenging datasets for semantic LiDAR point cloud segmentation and show that leveraging reformulated 3D point-based operations with 2D image-based operations achieves very good results for all four datasets.
BibTeX
@article{li2025rethinking,
  author = {Shijie Li and Yun Liu and Jüergen Gall},
  title = {{Rethinking 3D LiDAR Point Cloud Segmentation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2021.3132836},
}
Figure