Projected-point-based Segmentation: A New Paradigm for LiDAR Point Cloud Segmentation

Shijie Li, Yun Liu, Juergen Gall

Abstract

Many point-based semantic segmentation methods have been designed for indoor scenarios, but they struggle if they are applied to point clouds that are captured by a LiDAR sensor in an outdoor environment. In order to make these methods more efficient and robust such that they can handle LiDAR data, we introduce the general concept of reformulating 3D point-based operations such that they can operate in the projection space. While we show by means of three point-based methods that the reformulated versions are between 300 and 400 times faster and achieve a higher accuracy, we furthermore demonstrate that the concept of reformulating 3D point-based operations allows to design new architectures that unify the benefits of point-based and image-based methods. As an example, we introduce a network that integrates reformulated 3D point-based operations into a 2D encoder-decoder architecture that fuses the information from different 2D scales. We evaluate the approach on four challenging datasets for semantic LiDAR point cloud segmentation and show that leveraging reformulated 3D point-based operations with 2D image-based operations achieves very good results for all four datasets.

Introduction

Semantic segmentation of LiDAR point clouds is a cornerstone of autonomous driving, but researchers have long faced a trade-off. Point-based methods (like PointNet++) are designed for 3D geometry and work well in small indoor spaces. However, they scale poorly to massive outdoor LiDAR scans because their computational cost explodes as the number of points increases. Furthermore, they struggle with the "vanishing density" of LiDAR—the fact that points get much sparser the further they are from the sensor.

Conversely, projection-based methods project the 3D scan onto a 2D "image" (spherical projection). These are incredibly fast because they use standard 2D convolutions, but they often discard the underlying 3D structural information, leading to lower accuracy. This paper asks: Can we keep the sophisticated mathematical operations of 3D point-based methods but execute them with the efficiency of 2D projections?

Methods: Reformulating 3D Operations

The core contribution is the reformulation of three major 3D operations to run directly in 2D projection space:

  1. Ray Sampling: Instead of using the computationally expensive Farthest Point Sampling (FPS), the authors propose sampling uniformly in the projection space. This naturally matches the sensor's physical distribution and reduces complexity from $O(N \log N)$ to $O(M)$.
  2. Reformulated Grouping: Instead of searching the entire 3D space for neighbors (ball queries), the method looks at a $k \times k$ window in the 2D projection and applies a 3D distance threshold to filter out points that are adjacent in the image but far apart in reality.
  3. UnPNet (Unprojection Network): A new architecture that integrates these reformulated 3D operations into a 2D encoder-decoder structure. It uses "Unprojection" layers to handle up-sampling and down-sampling while preserving 3D features.
Comparison between Farthest Point Sampling (FPS) and the proposed Ray Sampling.

The network also employs Edge Supervision to ensure sharper boundaries between different semantic objects, such as a pedestrian standing next to a car.

Key Insights and Results

The authors evaluated their approach on four major benchmarks: SemanticKITTI, SemanticPOSS, nuScenes, and Pandaset.

  • The Efficiency Paradox: Surprisingly, the reformulated versions of PointNet++, SpiderCNN, and PointConv didn't just get faster—they got more accurate. By operating in projection space, the models were better able to handle the varying density of LiDAR data.
  • Sampling Matters: Farthest Point Sampling (FPS) is the standard in 3D research, but it is actually detrimental for LiDAR as it tends to oversample distant, noisy outliers. Ray Sampling is not only faster but provides a more representative distribution of the scene.
  • Performance: Reformulated baselines were 300 to 400 times faster than their original 3D counterparts. On SemanticKITTI, the reformulated versions improved mean Intersection over Union (mIoU) by 58% to 68%.
LiDAR point cloud segmentation results visualized in both 3D space and 2D projection.

The proposed UnPNet achieved top-tier results across all four datasets. For instance, on SemanticKITTI, it reached 54.6 mIoU, significantly outperforming previous projection-based leaders like RangeNet53.

Conclusion

This "reformulation paradigm" demonstrates that high-accuracy 3D logic can be achieved at 2D speeds. By bridging the gap between point-based and image-based methods, UnPNet captures multi-scale information more effectively than either paradigm could alone. Future work could involve applying these principles to even more complex architectures, such as point-voxel methods, to make them viable for real-time use in autonomous vehicle hardware.

Summary updated

Abstract

Many point-based semantic segmentation methods have been designed for indoor scenarios, but they struggle if they are applied to point clouds that are captured by a LiDAR sensor in an outdoor environment. In order to make these methods more efficient and robust such that they can handle LiDAR data, we introduce the general concept of reformulating 3D point-based operations such that they can operate in the projection space. While we show by means of three point-based methods that the reformulated versions are between 300 and 400 times faster and achieve a higher accuracy, we furthermore demonstrate that the concept of reformulating 3D point-based operations allows to design new architectures that unify the benefits of point-based and image-based methods. As an example, we introduce a network that integrates reformulated 3D point-based operations into a 2D encoder-decoder architecture that fuses the information from different 2D scales. We evaluate the approach on four challenging datasets for semantic LiDAR point cloud segmentation and show that leveraging reformulated 3D point-based operations with 2D image-based operations achieves very good results for all four datasets.
BibTeX
@article{li2025projected,
  author = {Shijie Li and Yun Liu and Juergen Gall},
  title = {{Projected-point-based Segmentation: A New Paradigm for LiDAR Point Cloud Segmentation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2021.3132836},
}
Figure