Multi Projection Fusion for Real-Time Semantic Segmentation of 3D LiDAR Point Clouds

Yara Ali Alnaggar, Mohamed Afifi, Karim Amer, Mohamed ElHelwWACV 2021

TL;DR

This paper introduces a novel Multi-Projection Fusion (MPF) framework that combines spherical and bird's-eye view projections of LiDAR point clouds to achieve more accurate and faster real-time semantic segmentation. By fusing information from these complementary views, MPF mitigates the information loss inherent in single-projection methods, leading to state-of-the-art performance on the SemanticKITTI dataset.

What Problem Does This Paper Address?

Autonomous systems, like self-driving cars and drones, rely heavily on LiDAR sensors to perceive their surroundings. Semantic segmentation, which assigns a label (e.g., "car," "road," "building") to each point in a 3D point cloud, is crucial for understanding these scenes and enabling safe navigation. However, processing dense 3D point clouds in real-time on resource-constrained mobile platforms (like onboard GPUs) is a significant challenge. Existing efficient methods often project 3D point clouds into 2D representations (like spherical or bird's-eye views) and use 2D Convolutional Neural Networks (CNNs). While this speeds up processing, it can lead to a loss of crucial 3D information, especially for distant objects. This paper aims to bridge this gap by developing a method that leverages multiple 2D projections to retain more information while still achieving real-time performance.

Methods

The core of the MPF framework is its ability to fuse information from two distinct 2D projections of the 3D point cloud:

  • Spherical Projection: This view maps the 3D point cloud onto a spherical image, similar to how a camera captures a panoramic view. The paper uses a modified MobileNetV2 as a lightweight encoder for a Fully Convolutional Neural Network (FCNN) to process this projection.
  • Bird's-Eye View Projection: This projection flattens the point cloud onto a 2D grid, resembling a top-down map. For this view, the authors employ a modified, lightweight UNet architecture.

The key novelty lies in the Multi-Projection Fusion itself. Each projection is processed independently by its respective FCNN. The resulting segmentation probabilities from both views are then fused using a soft voting mechanism. This fusion process combines the strengths of both projections, compensating for the weaknesses of each individual view. A post-processing step refines the segmentation by considering the spatial relationships and distances of points within a local window.

Datasets

The proposed MPF framework is trained and evaluated on the SemanticKITTI dataset. This dataset is a widely used benchmark for 3D semantic segmentation, containing approximately 43,000 360° LiDAR scans. The authors used sequence 08 for validation and trained on the remaining 10 sequences.

Key Insights and Ideas

  • Complementary Nature of Projections: Spherical and bird's-eye views capture different aspects of the 3D scene. Spherical views can struggle with distant objects due to projection distortion, while bird's-eye views offer a more uniform spatial resolution, making them better for objects at varying distances. Fusing them effectively leverages these complementary strengths.
  • Information Preservation through Fusion: By combining information from multiple views, the framework significantly reduces the information loss that occurs when relying on a single projection. This leads to more accurate segmentation, especially for challenging classes or objects at different ranges.
  • Efficiency through Lightweight Architectures: The use of efficient CNN backbones like MobileNetV2 and a streamlined UNet architecture for each projection is crucial for achieving real-time performance on mobile GPUs.
  • Soft Voting for Robust Fusion: The soft voting mechanism for fusing segmentation results allows for a more nuanced combination of probabilities from each view, leading to improved overall accuracy.

Key Results

The MPF framework demonstrates impressive results on the SemanticKITTI dataset:

  • Accuracy: It achieved a mean Intersection-over-Union (mIoU) of 55.5%, outperforming state-of-the-art projection-based methods like RangeNet++ (53.6%) and PolarNet (54.3%) on the test set.
  • Speed: MPF is significantly faster than its competitors, achieving 1.6x the speed of RangeNet++ and 3.1x the speed of PolarNet.
  • Qualitative Improvements: Visual results show MPF correctly segmenting challenging objects like 'person' and 'bicyclist' (classes that are often underrepresented or difficult to segment) and accurately identifying 'other-vehicle' objects that were missed by RangeNet53++.
  • Data Augmentation Impact: Data augmentation improved validation mIoU by 2.7% for the spherical view, 7.7% for the bird's-eye view, and an overall 7.2% for the fused model.

Limitations and Future Work

  • Limitations: While MPF significantly improves upon single-projection methods, it still relies on 2D projections, which inherently involve some information loss compared to direct 3D processing. The paper also notes that 3D-methods (though not discussed in detail for performance comparison) achieve higher mIoU scores but at a significant computational cost.
  • Future Work: The authors suggest several promising directions:
  • Combining both projections into a single, unified multi-view model.
  • Investigating the use of more than two projections to further enhance performance.
  • Exploring other efficient network architectures for the individual branches.

Summary updated

BibTeX
@inproceedings{alnaggar2021multi,
  author = {Yara Ali Alnaggar and Mohamed Afifi and Karim Amer and Mohamed ElHelw},
  title = {{Multi Projection Fusion for Real-Time Semantic Segmentation of 3D LiDAR Point Clouds}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure