MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detection

Michelle Adeline, Junn Yong Loo, Vishnu Monn Baskaran

Abstract

Multi-view 3D object detection is a crucial component of autonomous driving systems. Contemporary query-based methods primarily depend either on dataset-specific initialization of 3D anchors, introducing bias, or utilize dense attention mechanisms, which are computationally inefficient and unscalable. To overcome these issues, we present MDHA, a novel sparse query-based framework, which constructs adaptive 3D output proposals using hybrid anchors from multi-view, multi-scale image input. Fixed 2D anchors are combined with depth predictions to form 2.5D anchors, which are projected to obtain 3D proposals. To ensure high efficiency, our proposed Anchor Encoder performs sparse refinement and selects the top-$k$ anchors and features. Moreover, while existing multi-view attention mechanisms rely on projecting reference points to multiple images, our novel Circular Deformable Attention mechanism only projects to a single image but allows reference points to seamlessly attend to adjacent images, improving efficiency without compromising on performance. On the nuScenes val set, it achieves 46.4% mAP and 55.0% NDS with a ResNet101 backbone. MDHA significantly outperforms the baseline where anchor proposals are modelled as learnable embeddings.

Introduction

In the domain of autonomous driving, accurately detecting 3D objects from a suite of 2D cameras is a fundamental challenge. Current state-of-the-art query-based detectors typically follow one of two paradigms: dense attention or sparse attention with fixed anchors. Dense attention models, such as the PETR series, attend to all image features globally, which incurs high computational costs and limits scalability to high-resolution inputs. Conversely, sparse attention models like Sparse4D rely on 3D anchors initialized through dataset-specific statistics (e.g., k-means clustering). This reliance introduces significant bias, potentially hindering generalization to environments with different spatial distributions.

Comparison of MDHA with PETR and Sparse4D series. MDHA utilizes hybrid anchors to generate adaptive 3D proposals directly from image features.

MDHA (Multi-Scale Deformable Transformer with Hybrid Anchors) is designed to bridge this gap. It provides a sparse, efficient architecture that generates adaptive 3D proposals directly from multi-view image features, eliminating the need for pre-calculated anchor statistics while maintaining high performance.

Methods

The MDHA framework introduces several architectural innovations to transition efficiently from 2D image space to 3D world space.

Hybrid Anchors and 2.5D Projection

Instead of utilizing learnable embeddings or fixed 3D points, MDHA leverages the inherent spatial information in image features. The system identifies 2D centers of image feature tokens and pairs them with depth predictions generated by a dedicated DepthNet. This combination forms "2.5D anchors." These anchors are then projected into 3D space using camera calibration parameters to create initial 3D proposals. This approach ensures that anchors are naturally distributed according to the visual evidence in the scene.

Anchor Encoder and Decoder

The Anchor Encoder refines these initial 3D proposals and selects the top-$k$ most promising candidates. This sparsification ensures that the subsequent decoder layers focus only on high-probability object locations, significantly reducing computational overhead.

The overall MDHA architecture, illustrating the flow from multi-view images through the Anchor Encoder and Decoder using Circular Deformable Attention.

Circular Deformable Attention (CDA)

A key limitation of standard deformable attention in multi-view settings is the rigid boundary between camera frames. MDHA introduces Circular Deformable Attention (CDA), which treats the surround-view cameras as a continuous 360° panoramic strip. By allowing reference points to "wrap around" and attend to features in adjacent views, CDA enables the model to effectively reason about large objects that span multiple camera frames, improving detection consistency across view boundaries.

Experimental Results

MDHA was evaluated on the nuScenes benchmark, which consists of 1,000 urban driving scenes. Using a ResNet101 backbone, MDHA achieved 46.4% mAP and 55.0% NDS on the validation set.

Efficiency and Generalization

  • Training Speed: MDHA is approximately 1.9x faster to train than dense-attention baselines like StreamPETR due to its sparse attention mechanism.
  • Inference Performance: The model achieves a real-time inference speed of 15.1 FPS on an NVIDIA RTX 4090.
  • Ablation Studies: Comparisons against a baseline using learnable embeddings (fixed anchors) demonstrate that the hybrid anchor approach significantly improves the model's ability to capture the true 3D layout of complex scenes, confirming the value of image-driven anchor initialization.

Conclusion

MDHA presents a robust solution for multi-view 3D object detection by combining the efficiency of sparse attention with the flexibility of image-based anchor generation. By utilizing hybrid 2.5D anchors and a novel circular attention mechanism, the framework achieves state-of-the-art performance while remaining computationally efficient and generalizable across different environments. Future work will explore incorporating full 3D box geometry into the initial proposals and further optimizing token processing for enhanced efficiency.

Summary updated

Abstract

Multi-view 3D object detection is a crucial component of autonomous driving systems. Contemporary query-based methods primarily depend either on dataset-specific initialization of 3D anchors, introducing bias, or utilize dense attention mechanisms, which are computationally inefficient and unscalable. To overcome these issues, we present MDHA, a novel sparse query-based framework, which constructs adaptive 3D output proposals using hybrid anchors from multi-view, multi-scale input. Fixed 2D anchors are combined with depth predictions to form 2.5D anchors, which are projected to obtain 3D proposals. To ensure high efficiency, our proposed Anchor Encoder performs sparse refinement and selects the top-k anchors and features. Moreover, while existing multi-view attention mechanisms rely on projecting reference points to multiple images, our novel Circular Deformable Attention mechanism only projects to a single image but allows reference points to seamlessly attend to adjacent images, improving efficiency without compromising on performance. On the nuScenes val set, it achieves 46.4% mAP and 55.0% NDS with a ResNet101 backbone. MDHA significantly outperforms the baseline, where anchor proposals are modelled as learnable embeddings.
BibTeX
@inproceedings{adeline2024mdha,
  author = {Michelle Adeline and Junn Yong Loo and Vishnu Monn Baskaran},
  title = {{MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=gLjpazMUrf},
}
Figure