Uplifting Range-View-based 3D Semantic Segmentation in Real-Time with Multi-Sensor Fusion

Shiqi Tan, Hamidreza Fazlali, Yixuan Xu, Yuan Ren, Bingbing Liu

TL;DR

This paper introduces LaCRange, a novel real-time 3D semantic segmentation method for autonomous vehicles that fuses LiDAR and camera data. It tackles the challenge of information loss and distortion in camera data when projected into a Range-View (RV) representation, achieving state-of-the-art results on the nuScenes benchmark.

What Problem Does This Paper Address?

Autonomous vehicles rely heavily on semantic segmentation to understand their surroundings for safe navigation. While LiDAR provides accurate geometric information, cameras offer rich textural details. Fusing these modalities is crucial, but projecting camera images onto LiDAR's Range-View (RV) representation, a common technique for efficiency, leads to significant distortion and loss of appearance information due to the sparse nature of LiDAR points. Existing methods struggle to balance information preservation with real-time performance, especially when dealing with multiple cameras. This paper aims to bridge this gap by proposing a method that effectively fuses multi-sensor data in real-time while mitigating the distortions introduced by RV projection.

Methods

LaCRange introduces three key novel components:

  • Distortion-Compensating Knowledge Distillation (DCKD): To address the distortion in projected RGB images, a knowledge distillation strategy is employed. A pre-trained "teacher" model, trained on original, undistorted RGB images, guides a "student" model that processes the distorted RV-projected RGB images. This transfer of knowledge helps the student model recover the lost appearance information.
  • Context-based Feature Fusion (CFF): This module enhances the fusion of LiDAR and camera features. It first concatenates features and then uses a "Modality Look-Up" (MLU) block to allow features from each modality to interact and retrieve complementary information. Two strategies for MLU are proposed: CBAM-based and Cross-Attention-based, with a combined approach offering a good balance of performance and efficiency.
  • Point Refinement Scheme: To close the performance gap between projected points and the full 3D point cloud, a two-stage refinement process is introduced:
  • Semantic-Range-Remission-based Feature Aggregation (SR2FA): This module aggregates features from neighboring pixels in the RV image, considering both semantic probability and range-remission information.
  • 3D Neighborhood-Aware Feature Augmentation (3D-NAFA): This module leverages a coarse-to-fine voxelization strategy to incorporate 3D neighborhood context, enriching point features with spatial relationships. This component is designed to be a portable and universal plug-in for other RV-based methods.

Datasets

The proposed method was evaluated on two large-scale autonomous driving datasets:

  • SemanticKITTI [5]: This dataset provides dense semantic segmentation labels for LiDAR sequences.
  • nuScenes [4]: This dataset offers a more comprehensive multimodal sensor suite, including LiDAR and multiple cameras, capturing diverse driving scenarios.

Key Insights and Ideas

  • Knowledge Distillation for Distortion Correction: Using a teacher-student setup with knowledge distillation is an effective way to "teach" a model to overcome the inherent distortions introduced by projecting 3D data onto 2D representations.
  • Contextual Fusion for Complementary Information: Simply concatenating features isn't enough. Allowing features from different modalities to "look up" and exchange information (as in the CFF module) leads to a more robust and comprehensive understanding of the scene.
  • Point Refinement is Crucial for RV-based Methods: RV-based methods, while efficient, inherently lose information. The proposed point refinement scheme significantly boosts performance by re-introducing geometric and semantic context that was lost during projection.
  • Universality of the Point Refinement: The 3D-NAFA module's effectiveness is demonstrated by its ability to improve the performance of other existing RV-based segmentation methods, highlighting its general applicability.

Key Results

  • LaCRange achieves state-of-the-art results on the nuScenes benchmark, outperforming other methods in terms of mIoU.
  • The method is real-time, with an inference speed of 50.1 ms on the nuScenes test set.
  • Ablation studies show that each proposed component (DCKD, CFF, SR2FA, 3D-NAFA) contributes to the overall performance improvement. The 3D-NAFA module provides the largest gain (4.5% mIoU increase on SemanticKITTI).
  • The point refinement module, when applied to other methods like SqueezeSegV3 and SalsaNext, leads to significant mIoU improvements (e.g., +5.3% for SalsaNext).

Limitations and Future Work

  • Computational Cost of Multi-Camera Fusion: While LaCRange processes a single RV image, the paper notes that methods like PMF become computationally expensive when dealing with multiple cameras to achieve full 360° coverage. This suggests future work could focus on more efficient multi-camera fusion strategies.
  • Performance on Small Objects: While LaCRange performs well, the paper mentions it is "much better or on par with PMF [9] in some small object classes with complex shapes such as bicycle, bicyclist, pole, and traffic-sign." This indicates that these classes, sensitive to RV projection distortions, could still be an area for further improvement.
  • Future Work: The authors suggest that the point refinement module can be easily attached to other RV-based segmentation methods, implying its potential for broader adoption and further research into its integration. The paper also implicitly points towards exploring more advanced fusion techniques and distortion mitigation strategies.

Summary updated

Abstract

Range-View(RV)-based 3D point cloud segmentation is widely adopted due to its compact data form. However, RV-based methods fall short in providing robust segmentation for the occluded points and suffer from distortion of projected RGB images due to the sparse nature of 3D point clouds. To alleviate these problems, we propose a new LiDAR and Camera Range-view-based 3D point cloud semantic segmentation method (LaCRange). Specifically, a distortion-compensating knowledge distillation (DCKD) strategy is designed to remedy the adverse effect of RV projection of RGB images. Moreover, a context-based feature fusion module is introduced for robust and preservative sensor fusion. Finally, in order to address the limited resolution of RV and its insufficiency of 3D topology, a new point refinement scheme is devised for proper aggregation of features in 2D and augmentation of point features in 3D. We evaluated the proposed method on large-scale autonomous driving datasets \ie SemanticKITTI and nuScenes. In addition to being real-time, the proposed method achieves state-of-the-art results on nuScenes benchmark
BibTeX
@inproceedings{tan2024uplifting,
  author = {Shiqi Tan and Hamidreza Fazlali and Yixuan Xu and Yuan Ren and Bingbing Liu},
  title = {{Uplifting Range-View-based 3D Semantic Segmentation in Real-Time with Multi-Sensor Fusion}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10610507},
}
Figure