3M3D: Multi-View, Multi-Path, Multi-Representation for 3D Object Detection

Jongwoo Park, Apoorv Singh, Varun Bankiti

Abstract

3D object detection from multi-view camera images is a critical task for autonomous driving, yet it remains challenging due to the inherent difficulty of inferring depth from 2D data. This paper presents 3M3D, a framework that employs Multi-view, Multi-path, and Multi-representation strategies to enhance feature extraction and object localization. By introducing Multi-view Axis Self-Attention and ROI Self-Attention, the model enables early-stage cross-camera communication and local feature refinement. Evaluated on the nuScenes benchmark, 3M3D achieves significant improvements, including a +2.1% NDS and +4.3% mAP increase over the DETR3D baseline, with particularly strong gains in detecting rare object categories.

Introduction

Detecting 3D objects such as vehicles and pedestrians using only 2D cameras is a complex inverse problem. While LiDAR provides direct depth measurements, camera-based systems are preferred for their cost-effectiveness and high resolution. However, most existing camera-based detectors extract features from each view independently, fusing them only at the detection head. This "late fusion" approach creates a bottleneck, as the backbone lacks the global context necessary to understand objects spanning multiple camera views or those that are partially occluded.

Methodology

The proposed 3M3D architecture addresses these challenges through three key Transformer-based innovations designed to facilitate early-stage feature interaction:

The 3M3D architecture overview, illustrating the integration of multi-view axis self-attention, ROI self-attention, and the hybrid MRQ system.
  1. Multi-view Axis Self-Attention: This layer allows the network to attend to pixels across different camera views at the same vertical height. By correlating horizontal "strips" across the panoramic view, the model gains global context efficiently, utilizing only 0.2% of the operations required by a full self-attention mechanism.
  2. ROI (Region of Interest) Self-Attention: To capture fine-grained local details, a window-based attention mechanism is employed. By shifting these windows diagonally, the network learns to better define object boundaries and handle varying shapes.
  3. Multi-representation Queries (MRQ): 3M3D utilizes a hybrid query system that combines 900 sparse floating queries with 500 dense Bird’s Eye View (BEV) queries. This dual approach leverages the flexibility of sparse queries and the comprehensive coverage of dense queries, ensuring higher recall for rare objects.

Experimental Results

The model was evaluated on the nuScenes dataset, which contains 1.4 million annotated 3D bounding boxes across 360-degree multi-view images. 3M3D demonstrated superior performance compared to the DETR3D baseline, achieving a +2.1% increase in nuScenes Detection Score (NDS) and a +4.3% increase in mean Average Precision (mAP).

Visualization of multi-view camera inputs and the horizontal windowing strategy employed in the multi-view axis self-attention mechanism.

The improvements were particularly pronounced for rare and difficult object categories. For instance, detection of trailers improved by 16.8%, bicycles by 10.5%, and trucks by 3.9%. These results suggest that the hybrid query system and global context are highly effective at addressing class imbalance and occlusion. While the model introduces a slight computational overhead—running at 31 images per second compared to the baseline's 35—the accuracy gains justify the trade-off for safety-critical applications.

Conclusion

3M3D significantly advances multi-view 3D object detection by enabling early-stage feature interaction and utilizing a robust hybrid query mechanism. Future work will focus on integrating temporal information and further optimizing the attention layers for real-time edge deployment in autonomous vehicles.

Summary updated

Abstract

3D visual perception tasks based on multi-camera images are essential for autonomous driving systems. The latest work in this field performs 3D object detection by leveraging multi-view images as an input and iteratively enhancing object queries (object proposals) by cross-attending multi-view features. However, individual backbone features are not updated with multi-view features, and it stays as a mere collection of the output of the single-image backbone network. Therefore we propose 3M3D: A Multi-view, Multi-path, Multi-representation for 3D Object Detection where we update both multi-view features and query features to enhance the representation of the scene in both fine panoramic view and coarse global view. Firstly, we update multi-view features by multi-view axis self-attention. It will incorporate panoramic information in the multi-view features and enhance understanding of the global scene. Secondly, we update multi-view features by self-attention of the Region of Interest (ROI) windows which encodes local finer details in the features. It will help exchange the information not only along the multi-view axis but also along the other spatial dimension. Lastly, we leverage the fact of the multi-representation of queries (MRQ) in different domains to further boost performance. Here we use sparse floating queries along with dense Bird’s Eye View (BEV) queries, which are later post-processed to filter duplicate detections. Moreover, we show performance improvements on the nuScenes benchmark dataset [1] on top of our baselines.
BibTeX
@inproceedings{park2023m,
  author = {Jongwoo Park and Apoorv Singh and Varun Bankiti},
  title = {{3M3D: Multi-View, Multi-Path, Multi-Representation for 3D Object Detection}},
  booktitle = {ICIP},
  year = {2023},
  url = {https://openreview.net/forum?id=kkfMPpe56U},
}
Figure