TL;DR
This paper introduces a novel Multi-Projection Fusion (MPF) framework that combines spherical and bird's-eye view projections of LiDAR point clouds to achieve more accurate and faster real-time semantic segmentation. By fusing information from these complementary views, MPF mitigates the information loss inherent in single-projection methods, leading to state-of-the-art performance on the SemanticKITTI dataset.
What Problem Does This Paper Address?
Autonomous systems, like self-driving cars and drones, rely heavily on LiDAR sensors to perceive their surroundings. Semantic segmentation, which assigns a label (e.g., "car," "road," "building") to each point in a 3D point cloud, is crucial for understanding these scenes and enabling safe navigation. However, processing dense 3D point clouds in real-time on resource-constrained mobile platforms (like onboard GPUs) is a significant challenge. Existing efficient methods often project 3D point clouds into 2D representations (like spherical or bird's-eye views) and use 2D Convolutional Neural Networks (CNNs). While this speeds up processing, it can lead to a loss of crucial 3D information, especially for distant objects. This paper aims to bridge this gap by developing a method that leverages multiple 2D projections to retain more information while still achieving real-time performance.
Methods
The core of the MPF framework is its ability to fuse information from two distinct 2D projections of the 3D point cloud:
- Spherical Projection: This view maps the 3D point cloud onto a spherical image, similar to how a camera captures a panoramic view. The paper uses a modified MobileNetV2 as a lightweight encoder for a Fully Convolutional Neural Network (FCNN) to process this projection.
- Bird's-Eye View Projection: This projection flattens the point cloud onto a 2D grid, resembling a top-down map. For this view, the authors employ a modified, lightweight UNet architecture.
The key novelty lies in the Multi-Projection Fusion itself. Each projection is processed independently by its respective FCNN. The resulting segmentation probabilities from both views are then fused using a soft voting mechanism. This fusion process combines the strengths of both projections, compensating for the weaknesses of each individual view. A post-processing step refines the segmentation by considering the spatial relationships and distances of points within a local window.
Datasets
The proposed MPF framework is trained and evaluated on the SemanticKITTI dataset. This dataset is a widely used benchmark for 3D semantic segmentation, containing approximately 43,000 360° LiDAR scans. The authors used sequence 08 for validation and trained on the remaining 10 sequences.
Key Insights and Ideas
- Complementary Nature of Projections: Spherical and bird's-eye views capture different aspects of the 3D scene. Spherical views can struggle with distant objects due to projection distortion, while bird's-eye views offer a more uniform spatial resolution, making them better for objects at varying distances. Fusing them effectively leverages these complementary strengths.
- Information Preservation through Fusion: By combining information from multiple views, the framework significantly reduces the information loss that occurs when relying on a single projection. This leads to more accurate segmentation, especially for challenging classes or objects at different ranges.
- Efficiency through Lightweight Architectures: The use of efficient CNN backbones like MobileNetV2 and a streamlined UNet architecture for each projection is crucial for achieving real-time performance on mobile GPUs.
- Soft Voting for Robust Fusion: The soft voting mechanism for fusing segmentation results allows for a more nuanced combination of probabilities from each view, leading to improved overall accuracy.
Key Results
The MPF framework demonstrates impressive results on the SemanticKITTI dataset:
- Accuracy: It achieved a mean Intersection-over-Union (mIoU) of 55.5%, outperforming state-of-the-art projection-based methods like RangeNet++ (53.6%) and PolarNet (54.3%) on the test set.
- Speed: MPF is significantly faster than its competitors, achieving 1.6x the speed of RangeNet++ and 3.1x the speed of PolarNet.
- Qualitative Improvements: Visual results show MPF correctly segmenting challenging objects like 'person' and 'bicyclist' (classes that are often underrepresented or difficult to segment) and accurately identifying 'other-vehicle' objects that were missed by RangeNet53++.
- Data Augmentation Impact: Data augmentation improved validation mIoU by 2.7% for the spherical view, 7.7% for the bird's-eye view, and an overall 7.2% for the fused model.
Limitations and Future Work
- Limitations: While MPF significantly improves upon single-projection methods, it still relies on 2D projections, which inherently involve some information loss compared to direct 3D processing. The paper also notes that 3D-methods (though not discussed in detail for performance comparison) achieve higher mIoU scores but at a significant computational cost.
- Future Work: The authors suggest several promising directions:
- Combining both projections into a single, unified multi-view model.
- Investigating the use of more than two projections to further enhance performance.
- Exploring other efficient network architectures for the individual branches.
Summary updated