TL;DR
This paper introduces DAOcc, a novel multi-sensor fusion framework for 3D occupancy prediction that achieves state-of-the-art accuracy and real-time performance. By leveraging 3D object detection as an auxiliary supervision signal and employing a deployment-friendly architecture, DAOcc significantly improves environmental understanding for autonomous driving and robotics.
What Problem Does This Paper Address?
Accurate 3D semantic occupancy prediction is vital for safe autonomous driving and robotics, as it involves understanding every occupied voxel in a scene and assigning it a semantic label. While multi-sensor fusion (combining LiDAR and cameras) offers improved accuracy and robustness, existing methods often rely on high-resolution images and computationally expensive networks, making them impractical for deployment on edge devices. Furthermore, many current fusion techniques focus on improving feature fusion itself, neglecting effective supervision strategies for these fused features. This paper aims to bridge this gap by proposing a more efficient and effective multi-modal fusion approach that is both accurate and deployable.
Methods
DAOcc introduces several key innovations:
- Deployment-Friendly Baseline: It starts with a simple yet effective baseline that uses a standard ResNet-50 backbone for image feature extraction and 3D sparse convolutions for point cloud processing. Image features are projected into 3D space and then fused with LiDAR features in a Bird's Eye View (BEV) representation.
- 3D Object Detection Supervision: A novel aspect is the use of a "detachable auxiliary detection head." This head is trained to perform 3D object detection on the fused BEV features. This auxiliary task helps to improve the discriminability of foreground objects and scene structures within the occupancy prediction framework, without adding computational overhead during inference.
- BEV View Range Extension (BVRE): To combat performance degradation from lower image resolutions, DAOcc extends the processing range of point clouds in the BEV space. This provides richer contextual information and helps to compensate for the reduced detail from lower-resolution images. The paper details a flexible pipeline for coordinate transformation and feature interpolation to ensure alignment with occupancy ground truth.
Datasets
The proposed DAOcc framework is evaluated on two prominent benchmarks:
- Occ3D-nuScenes: A large-scale dataset for 3D occupancy prediction, featuring 700 training scenes and 150 validation scenes. It includes synchronized RGB images from six cameras and LiDAR point clouds, with a perception range of [-40m, 40m] in X and Y.
- Occ3D-Waymo: Another large-scale dataset with 798 training and 202 validation scenes, sharing similar perception ranges and voxel sizes with Occ3D-nuScenes. It utilizes data from five cameras and multiple LiDARs.
Key Insights and Ideas
- 3D Object Detection as a Powerful Auxiliary Task: The paper demonstrates that using 3D object detection as an auxiliary supervision signal significantly boosts performance, particularly for foreground object categories. This suggests that leveraging object-level understanding can directly improve dense scene prediction.
- Efficiency Through Simplicity: By avoiding complex modules like deformable attention and relying on a standard ResNet-50 backbone with practical input resolutions, DAOcc achieves a remarkable balance between performance and computational efficiency.
- Spatial Context is Key: The BVRE strategy highlights the importance of a wider spatial context in BEV for occupancy prediction, especially when dealing with lower-resolution inputs. Expanding the BEV range, even with minimal computational increase for LiDAR processing, yields tangible accuracy improvements.
- Unified Framework Potential: The auxiliary detection branch achieves competitive 3D object detection performance, suggesting DAOcc's potential as a foundation for a multi-task 3D perception framework.
Key Results
- State-of-the-Art Performance: DAOcc achieves new state-of-the-art results on both Occ3D-nuScenes and Occ3D-Waymo benchmarks.
- On Occ3D-nuScenes (with camera mask), it achieves 54.33 mIoU using a ResNet-50 backbone and 256×704 input resolution.
- It outperforms the previous state-of-the-art SDGOCC by 2.67 mIoU while maintaining a higher FPS.
- Real-time Inference: With TensorRT optimization on an NVIDIA RTX 4090 GPU, DAOcc achieves 104.9 FPS while maintaining 54.2 mIoU.
- Robustness: Qualitative results show DAOcc's strong performance in challenging low-illumination and rainy scenarios, accurately reconstructing occupancy despite visual occlusions.
- Auxiliary Detection Performance: The auxiliary detection branch achieves 59.4 mAP and 64.3 NDS on the nuScenes validation set.
Limitations and Future Work
- Detection Supervision Imbalance: The authors note that while auxiliary detection supervision enhances foreground object representation, it can slightly weaken the representation of background categories. They suggest that additional BEV segmentation supervision could help balance this.
- Robustness Design: While DAOcc shows inherent robustness, it was not specifically designed for it. The authors suggest that increasing training data and incorporating specific data augmentation strategies for sensor failures could further enhance robustness.
- Multi-Task Extension: The strong performance of the auxiliary detection branch suggests future work in developing DAOcc into a more comprehensive multi-task 3D perception framework.
Summary updated