TL;DR
This paper introduces PromptDet, a novel framework that efficiently integrates LiDAR data into camera-based 3D object detection using a "prompt learning" approach. PromptDet achieves significant performance improvements in multi-modal detection and even enhances camera-only detection without substantial computational overhead, offering a flexible solution for real-world autonomous driving scenarios.
What Problem Does This Paper Address?
Autonomous driving systems rely heavily on accurate 3D object detection to understand their surroundings. While camera-based systems are cost-effective and scalable, they inherently struggle with depth estimation and precise 3D perception due to the limitations of visual data. LiDAR sensors excel at providing accurate depth information but are more expensive and can be less effective in adverse weather conditions.
Existing multi-modal approaches often combine camera and LiDAR data but tend to be computationally expensive and memory-intensive, requiring complex network architectures and lengthy training times. Furthermore, these methods can suffer performance degradation when LiDAR data is unavailable. Knowledge distillation (KD) offers a way to improve camera-only detectors by learning from a pre-trained multi-modal "teacher" model, but this process is also laborious and requires a separate teacher model. This paper addresses the need for a lightweight, efficient, and flexible 3D object detection framework that can leverage LiDAR data to boost performance while remaining effective even when LiDAR is absent.
Methods
PromptDet introduces a "LiDAR-assisted prompter" that acts as a plug-and-play module for existing camera-based 3D object detectors, inspired by prompt learning in natural language processing and computer vision. The core idea is to inject complementary geometric information from LiDAR into the camera pipeline with minimal additional parameters.
The framework consists of two main components:
- Adaptive Hierarchical Aggregation (AHA): This module fuses LiDAR point cloud features and camera-derived pseudo-voxel features at multiple scales. It adaptively aggregates these multi-scale fusion features to create a richer representation of the 3D scene.
- Cross-Modal Knowledge Injection (CMKI): This module injects the fused multi-modal information into the camera-only detector's pipeline. To prevent the camera-only branch from being negatively impacted by the fusion process during training, CMKI employs a "detaching" mechanism, stopping the backward gradient from the fused features to the camera-only branch. This ensures that the camera-only detector learns from the LiDAR prompts without compromising its independent performance.
PromptDet also utilizes a "hybrid supervision strategy" where both the fused multi-modal features and the camera-only features are supervised by the ground truth. This allows for efficient single-stage training. A key feature is the "LiDAR modality switch," which enables PromptDet to function as a multi-modal detector when LiDAR is available and seamlessly revert to a camera-only detector with enhanced performance when LiDAR is absent.
Datasets
The proposed PromptDet framework is evaluated on the nuScenes dataset. This dataset is a standard benchmark for autonomous driving, featuring synchronized LiDAR point clouds and images from six surrounding cameras across 1,000 driving scenarios.
Key Insights and Ideas
- Prompt Learning for 3D Detection: The paper successfully demonstrates that the prompt learning paradigm, successful in 2D vision and NLP, can be effectively adapted to 3D object detection to inject auxiliary modality information (LiDAR) into a primary modality (camera) with minimal overhead.
- Flexibility of LiDAR as a "Prompt": LiDAR data is treated not as a primary input that dictates the entire architecture, but as a flexible "prompt" that enhances an existing camera-based detector. This allows for graceful degradation when LiDAR is unavailable.
- Efficient Multi-Modal Fusion: AHA provides a novel way to fuse features from different modalities at multiple scales, creating a more comprehensive understanding of the 3D environment without requiring a massive increase in model complexity.
- CMKI for Enhanced Camera-Only Performance: The CMKI module, with its detaching mechanism, allows the camera-only detector to benefit from LiDAR information during training without being overly dependent on it, leading to improved camera-only performance even when LiDAR is not used during inference.
Key Results
- Multi-modal Performance Boost: When used as a multi-modal detector (PromptDet-LC), the framework significantly improves performance over camera-only baselines. For example, when applied to BEVDet, it achieves up to a 22.8% increase in mAP and 21.1% in NDS with fewer than 2% additional parameters compared to the camera-only baseline.
- Camera-Only Enhancement: Even when LiDAR is unavailable during inference, PromptDet (PromptDet-C) enhances the performance of camera-only detectors. For BEVDet, this resulted in up to a 2.4% mAP and 4.0% NDS improvement.
- Efficiency: PromptDet is lightweight. The addition of the LiDAR-assisted prompter adds minimal parameters (around 1-2%) and has a negligible impact on inference time for the camera-only version. PromptDet-LC is also shown to be significantly faster than other multi-modal methods like BEVFusion while achieving better performance.
Limitations and Future Work
- LiDAR Dependency: While PromptDet is designed to be flexible, its multi-modal performance is still dependent on the availability and quality of LiDAR data. The paper notes that performance can degrade if LiDAR data is unavailable, though it still outperforms the camera-only baseline.
- Future Directions: The authors suggest that the PromptDet paradigm is compatible with various camera-based detectors and can be extended to other multi-camera perception tasks such as occupancy prediction and BEV segmentation. They also highlight the potential for further exploration of prompt learning in 3D perception.
Summary updated