LiDAR point cloud segmentation (PCS) is the "eyes" of a self-driving car, allowing it to distinguish a cyclist from a sidewalk in real-time. To do this efficiently, researchers often project 3D point clouds into 2D "range images." While this makes processing lightning-fast, it introduces a frustrating problem: missing values.
A new paper, "Filling Missing Values Matters for Range Image-Based Point Cloud Segmentation," argues that these "holes" in the data are more than just a nuisance—they are a major bottleneck for model performance. By rethinking how we project data and fill these gaps, the authors achieve state-of-the-art results with impressive efficiency.
TL;DR
The authors identify that standard projection methods and motion compensation (deskewing) create artificial "holes" in LiDAR range images, breaking object shapes. They propose Scan Unfolding++ (SU++) to project data more cleanly, a range-dependent KNN interpolation (KNNI) to fill remaining gaps, and a high-performance network called FMVNet that beats existing models in the speed-accuracy trade-off.
What Problem Does This Paper Address?
To process massive outdoor point clouds, models often convert 3D points into a 2D grid (a range image). Think of this like unwrapping a 360-degree panoramic photo where each pixel represents the distance to an object.
However, these images are often riddled with black pixels (missing values). The authors point to three culprits:
- Projection Overlap: Standard spherical projections cause scan lines to overlap, especially when lasers aren't perfectly spaced.
- The "Deskewing" Paradox: To account for a car's motion, we "deskew" scans. While this is great for 3D accuracy, it creates horizontal gaps when projected into 2D.
- Sensor Physics: Sometimes a laser beam just doesn't come back—it hits a black car that absorbs the light or flies off into the infinite sky.
These holes make objects look "broken" (e.g., a bicyclist appearing as a floating torso and a wheel), making it incredibly hard for a neural network to learn coherent geometric shapes.
Methods
The authors tackle the problem with a three-pronged approach:
1. Scan Unfolding++ (SU++)
Instead of standard spherical projection, they use "Scan Unfolding." The "++" version introduces a "skewing" method. Essentially, they temporarily "un-do" the motion compensation during the projection phase to ensure points land in a continuous grid, then use a Look-Up Table (LUT) to map them back. This drastically reduces the number of artificial holes created during preprocessing.
2. Range-Dependent KNN Interpolation (KNNI)
For the holes that remain, the authors use a simple but clever interpolation. For any "invalid" pixel, they look at its horizontal neighbors. Crucially, they fill the hole using the neighbor with the smallest range value (the one closest to the sensor). This ensures that foreground objects (like a pedestrian) stay visible and aren't "eaten" by the background (like a distant wall).
3. FMVNet (Filling Missing Values Network)
They designed a new backbone based on the ConvNeXt architecture. Most image-based networks use a "Stem" module that aggressively shrinks the image (e.g., a 4x4 stride). Since LiDAR range images are already short (often only 64 pixels high), this kills the resolution. FMVNet uses a 1x1 convolution in the stem to preserve every bit of vertical detail.
Datasets
The researchers evaluated their methods on three heavy-hitters in the autonomous driving world:
- SemanticKITTI: The gold standard for large-scale outdoor LiDAR segmentation.
- SemanticPOSS: A dataset specifically rich in dynamic instances like people and riders.
- nuScenes: A massive, diverse dataset for urban navigation.
Key Insights and Ideas
- Preprocessing is as important as the model: The authors showed that simply changing how you project the data (SU++) can boost the performance of existing models (like RangeNet53++ or FIDNet) without changing a single line of the model's code.
- The "Minimum Range" Rule: In interpolation, picking the closest neighbor is a powerful heuristic. It mimics how human vision works—closer objects occlude further ones.
- Resolution is Precious: In 2D image tasks, we often downsample to save compute. In LiDAR, where the vertical resolution is already tiny (64 or 128 "pixels"), downsampling is a "performance killer." Preserving resolution in the early layers of the network is vital.
Key Results
- Performance Boost: On SemanticKITTI, applying SU++ and KNNI to the baseline RangeNet53++ boosted its mean Intersection over Union (mIoU) from 61.5% to 64.4%.
- State-of-the-Art Trade-off: Their FMVNet achieved 69.0% mIoU on the SemanticKITTI validation set.
- Real-time Ready: They introduced Fast FMVNet, which maintains a high accuracy (67.4% mIoU) while running at 48 FPS on an NVIDIA A100—well above the 10-20 FPS requirement for real-time robotics.
Limitations and Future Work
While highly effective, the authors note a few areas for improvement:
- Manual Intervention: Currently, SU++ requires users to know the "laser ID" or "ring number" for each point. While modern sensors provide this, older datasets might require manual labeling or estimation.
- Hyperparameter Tuning: The authors used a fixed learning rate and standard loss functions; they suggest that a more exhaustive grid search for these parameters could squeeze even more performance out of the model.
- Beyond Segmentation: The authors believe these "hole-filling" techniques could be applied to other tasks like object detection or LiDAR-based SLAM (Simultaneous Localization and Mapping).
Summary updated