Abstract
3D object detection from multi-view camera images is a critical task for autonomous driving, yet it remains challenging due to the inherent difficulty of inferring depth from 2D data. This paper presents 3M3D, a framework that employs Multi-view, Multi-path, and Multi-representation strategies to enhance feature extraction and object localization. By introducing Multi-view Axis Self-Attention and ROI Self-Attention, the model enables early-stage cross-camera communication and local feature refinement. Evaluated on the nuScenes benchmark, 3M3D achieves significant improvements, including a +2.1% NDS and +4.3% mAP increase over the DETR3D baseline, with particularly strong gains in detecting rare object categories.
Introduction
Detecting 3D objects such as vehicles and pedestrians using only 2D cameras is a complex inverse problem. While LiDAR provides direct depth measurements, camera-based systems are preferred for their cost-effectiveness and high resolution. However, most existing camera-based detectors extract features from each view independently, fusing them only at the detection head. This "late fusion" approach creates a bottleneck, as the backbone lacks the global context necessary to understand objects spanning multiple camera views or those that are partially occluded.
Methodology
The proposed 3M3D architecture addresses these challenges through three key Transformer-based innovations designed to facilitate early-stage feature interaction:
- Multi-view Axis Self-Attention: This layer allows the network to attend to pixels across different camera views at the same vertical height. By correlating horizontal "strips" across the panoramic view, the model gains global context efficiently, utilizing only 0.2% of the operations required by a full self-attention mechanism.
- ROI (Region of Interest) Self-Attention: To capture fine-grained local details, a window-based attention mechanism is employed. By shifting these windows diagonally, the network learns to better define object boundaries and handle varying shapes.
- Multi-representation Queries (MRQ): 3M3D utilizes a hybrid query system that combines 900 sparse floating queries with 500 dense Bird’s Eye View (BEV) queries. This dual approach leverages the flexibility of sparse queries and the comprehensive coverage of dense queries, ensuring higher recall for rare objects.
Experimental Results
The model was evaluated on the nuScenes dataset, which contains 1.4 million annotated 3D bounding boxes across 360-degree multi-view images. 3M3D demonstrated superior performance compared to the DETR3D baseline, achieving a +2.1% increase in nuScenes Detection Score (NDS) and a +4.3% increase in mean Average Precision (mAP).
The improvements were particularly pronounced for rare and difficult object categories. For instance, detection of trailers improved by 16.8%, bicycles by 10.5%, and trucks by 3.9%. These results suggest that the hybrid query system and global context are highly effective at addressing class imbalance and occlusion. While the model introduces a slight computational overhead—running at 31 images per second compared to the baseline's 35—the accuracy gains justify the trade-off for safety-critical applications.
Conclusion
3M3D significantly advances multi-view 3D object detection by enabling early-stage feature interaction and utilizing a robust hybrid query mechanism. Future work will focus on integrating temporal information and further optimizing the attention layers for real-time edge deployment in autonomous vehicles.
Summary updated