Abstract
Point cloud segmentation (PCS) is essential for environmental understanding in robotics and autonomous driving. While range image-based models are computationally efficient, the projection of 3D point clouds onto 2D grids inevitably leads to information loss due to dropped points. This creates a theoretical performance ceiling and makes it challenging to correctly classify points that share image coordinates but belong to different classes. We propose a trainable Pointwise Decoder Module (PDM) as a post-processing approach that gathers weighted features from neighbors to refine per-point predictions. Additionally, we introduce Virtual Range Image-guided Copy-Rotate-Paste (VRCrop), a data augmentation strategy that eliminates artifacts and maintains stable point counts. Our methods consistently improve the performance of range image-based models on the SemanticKITTI, SemanticPOSS, and nuScenes datasets.
Introduction
Autonomous vehicles and robots often utilize range images as a dense proxy for 3D LiDAR data because 2D convolutional neural networks (CNNs) offer superior inference speeds. However, this efficiency comes at the cost of discretization error. When a 3D point cloud is projected onto a 2D grid, multiple points often map to the same pixel coordinate. Standard models typically retain only the nearest point, discarding the rest. During inference, these "dropped" points are assigned the same label as the retained point, which leads to errors if they belong to different objects. Existing solutions like K-Nearest Neighbor (KNN) search are often non-trainable and fail to capture complex local geometries.
Methodology
Pointwise Decoder Module (PDM)
To overcome the limitations of static post-processing, we developed the Pointwise Decoder Module (PDM). PDM is an end-to-end trainable module that plugs into the end of a range-image model. It uses a range-image-guided KNN search to efficiently identify neighbors in 3D space.
As illustrated in the pipeline, the PDM processes features from the range image alongside the original point cloud. By utilizing a local feature extraction module with attentive weights—calculated from relative positions and edge features—the PDM can "re-predict" labels for every point, including those dropped during the initial projection.
Virtual Range Image-guided Copy-Rotate-Paste (VRCrop)
Standard "copy-paste" augmentation often creates "ghosting" artifacts where new points overlap existing ones, or causes the point count to explode. VRCrop addresses this by projecting points onto a virtual range image. When an object is pasted into a scene, VRCrop identifies and deletes the original points at those coordinates first. This ensures the model learns from physically plausible data and maintains a stable point count during training.
Experimental Results
We evaluated the PDM and VRCrop modules across several backbones, including RangeNet53++, FIDNet, and CENet, on three major outdoor LiDAR datasets.
- Performance Boost: The proposed modules consistently improved mean Intersection-over-Union (mIoU). For instance, Fast FMVNet saw an mIoU increase from 66.03% to 68.55% on SemanticKITTI. On nuScenes, the PDM helped models approach and break through the theoretical mIoU limit of ~91.5% imposed by projection loss.
- Real-Time Viability: By optimizing the neighbor search through range-image guidance, the PDM remains highly efficient. Our optimized model, Fast FMVNetv2, achieved a frame rate of 26.5 FPS on an NVIDIA A100 GPU.
- State-of-the-Art Comparison: On the SemanticKITTI test set, our method outperformed several recent architectures, particularly in challenging classes such as "Bicycle," "Person," and "Car."
Conclusion
The Trainable Pointwise Decoder Module (PDM) effectively addresses the information loss inherent in range-image-based point cloud segmentation. By making the refinement process trainable and improving data quality through VRCrop, we achieve significant accuracy gains without sacrificing real-time performance. Future work will focus on further refining VRCrop to better simulate LiDAR occlusion and shadowing effects.
Summary updated