Introduction
LiDAR perception is a cornerstone of autonomous driving, traditionally divided into three distinct tasks: 3D object detection, semantic segmentation, and panoptic segmentation. While specialized models for each task have reached high levels of maturity, deploying them simultaneously in a vehicle is computationally expensive and fails to leverage the inherent synergies between identifying objects and labeling points. LidarMultiNet addresses this by proposing a unified, multi-task architecture that achieves state-of-the-art performance across all three domains using only LiDAR data.
Methodology: A Unified Architecture
LidarMultiNet is built upon a shared 3D voxel-based encoder-decoder (U-Net) backbone. This shared feature extractor allows the model to learn a rich representation of the environment that benefits all downstream tasks simultaneously.
Global Context Pooling (GCP)
A significant challenge in 3D sparse convolutions is their limited receptive field. To overcome this, the authors introduced the Global Context Pooling (GCP) module. GCP bridges the encoder and decoder by projecting 3D features into a 2D Bird’s Eye View (BEV), applying a multi-scale 2D CNN to capture long-range spatial context, and then re-projecting these features back into the 3D space. This allows the network to "see" large-scale structures like buildings or long stretches of road, which is critical for accurate semantic labeling.
Multi-Task Heads and Refinement
The network branches into specialized heads at the end of the pipeline:
- Semantic Head: A 3D head that predicts class labels for every point in the cloud.
- Detection Head: A 2D BEV branch utilizing an anchor-free CenterPoint-style head to generate 3D bounding boxes.
- Panoptic Refinement: To generate high-quality panoptic results, a second-stage point-based refinement module ensures consistency between detection and segmentation, "cleaning up" labels within detected object instances.
Benchmarks and Results
The model was rigorously tested on the Waymo Open Dataset (WOD) and nuScenes, two of the most challenging benchmarks in the field. LidarMultiNet demonstrated that multi-task training provides a better result than training tasks in isolation.
- Waymo Open Dataset: LidarMultiNet secured 1st place in the 2022 3D Semantic Segmentation Challenge with 71.13 mIoU. It also established a new state-of-the-art for single-model 3D detection with 76.35 mAPH L2.
- nuScenes: The model outperformed all previously published methods across all three tasks, notably achieving 81.4% Panoptic Quality (PQ) and 81.4% mIoU.
Ablation studies confirmed that the GCP module alone provided a nearly 1% mIoU boost, proving that capturing global context is essential for high-fidelity LiDAR perception.
Conclusion
LidarMultiNet demonstrates that a single, well-designed network can match or exceed the performance of specialized single-task models. By utilizing Global Context Pooling and a unified backbone, it offers a more efficient and accurate solution for autonomous vehicle perception. While currently LiDAR-only, future iterations could integrate camera data or extend the framework to include temporal tasks like motion forecasting.
Summary updated