MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detection

Michelle Adeline, Junn Yong Loo, Vishnu Monn Baskaran

MDHA: Bridging the Gap in Multi-View 3D Object Detection

3D object detection is the "eyes" of an autonomous vehicle, but doing it with cameras alone (without expensive LiDAR) is a massive computational challenge. Current state-of-the-art methods usually fall into two camps: those that use dense attention (accurate but slow and unscalable) and those that use sparse queries (efficient but often biased by dataset-specific initializations).

In their latest paper, researchers from Monash University Malaysia introduce MDHA (Multi-Scale Deformable Transformer with Hybrid Anchors), a framework that combines the best of both worlds. By rethinking how 3D anchors are initialized and how attention is paid across multiple camera views, MDHA achieves high performance with impressive efficiency.

TL;DR

MDHA is a sparse, query-based 3D detection framework that replaces static, biased anchor points with "hybrid anchors" derived from image-level depth predictions. It introduces a Circular Deformable Attention (CDA) mechanism that treats multi-view camera inputs as a continuous 360° panorama, allowing the model to detect objects seamlessly even when they are split across two different camera frames.

What Problem Does This Paper Address?

Most query-based 3D detectors rely on one of two strategies:

  1. Dense Attention (e.g., PETR): These models look at everything at once. While accurate, they are computationally heavy and don't scale well with high-resolution images.
  2. Sparse Attention (e.g., Sparse4D): These models only look at specific "anchor points." However, these anchors are often initialized using k-means clustering on the training set. This introduces a "dataset bias"—the model expects objects to be where they usually are in the training data, which might not hold true in new, diverse driving environments.

MDHA addresses this by creating adaptive proposals. Instead of guessing where objects might be, it uses 2D image features and predicted depth to "project" 3D anchors into the scene, ensuring the model's focus is always grounded in the actual visual evidence.

Methods

The MDHA architecture consists of three main innovations:

1. Hybrid Anchors via DepthNet

Instead of starting with random 3D points, MDHA starts with the 2D center coordinates of every image feature token. A module called DepthNet predicts the depth ($z$) for these tokens. By combining 2D coordinates with predicted depth, the model creates 2.5D anchors, which are then projected into 3D space using camera matrices. This reduces the search space for the model from a 3D volume ($\mathbb{R}^3$) to a 1D depth line ($\mathbb{R}^1$).

2. The Anchor Encoder

Because multi-scale, multi-view inputs generate a massive number of tokens, MDHA uses a shallow Anchor Encoder to refine these proposals and select only the "top-k" most promising anchors to pass into the main decoder. This keeps the system sparse and fast.

3. Circular Deformable Attention (CDA)

Standard attention mechanisms often struggle with "boundary" issues—if a car is half in the 'Front' camera and half in the 'Front-Right' camera, the model sees them as two separate entities. MDHA solves this by concatenating all camera views into a single horizontal strip. The CDA mechanism uses a "wrap-around" (modulo) logic, treating the last camera and the first camera as neighbors. This allows reference points to attend to adjacent images seamlessly, mimicking a true 360° panoramic view.

Datasets

The model was trained and evaluated on the nuScenes dataset, a large-scale autonomous driving benchmark featuring 1,000 scenes with full 360° sensor coverage. The authors used the standard metrics: mean Average Precision (mAP) and nuScenes Detection Score (NDS).

Key Insights and Ideas

  • Depth as a Bridge: Using depth to lift 2D features into 3D is more robust than learnable embeddings. It allows the model to generalize to new environments because it relies on geometry rather than memorized patterns.
  • The Panorama Advantage: Treating multi-view images as a circular ring (CDA) is a "simple but elegant" fix for the multi-camera overlap problem. It improves performance without adding significant computational overhead.
  • Efficiency Matters: By using a sparse approach and a single-window temporal memory, MDHA avoids the "memory explosion" seen in dense models like PETR.

Key Results

  • Performance: On the nuScenes validation set, MDHA (with a ResNet101 backbone) achieved 46.4% mAP and 55.0% NDS.
  • Speed: MDHA is significantly faster than its competitors. Specifically, it trains 1.9x faster than StreamPETR while maintaining comparable or better accuracy.
  • Generalization: MDHA outperformed a baseline using "learnable anchors" by 7.1 percentage points in mAP, proving that adaptive, depth-based anchors are far superior to static ones.
  • Inference: The "fixed depth" version of the model achieved an inference speed of 15.1 FPS on an RTX 4090, making it highly viable for real-time applications.

Limitations and Future Work

While MDHA is a step forward, the authors note a few areas for improvement:

  • Token Sparsification: Currently, the encoder still processes a large number of tokens before filtering. Further sparsifying the input tokens could boost efficiency even more.
  • Full 3D Anchors: The current model focuses primarily on 3D centers. Incorporating full 3D box information (width, length, height) earlier in the proposal stage could improve detection of complex object shapes.
  • Temporal Window: The model currently uses a relatively small temporal window (last 4 frames). Expanding this could help in scenarios with long-term occlusions.

Summary updated

Abstract

Multi-view 3D object detection is a crucial component of autonomous driving systems. Contemporary query-based methods primarily depend either on dataset-specific initialization of 3D anchors, introducing bias, or utilize dense attention mechanisms, which are computationally inefficient and unscalable. To overcome these issues, we present MDHA, a novel sparse query-based framework, which constructs adaptive 3D output proposals using hybrid anchors from multi-view, multi-scale image input. Fixed 2D anchors are combined with depth predictions to form 2.5D anchors, which are projected to obtain 3D proposals. To ensure high efficiency, our proposed Anchor Encoder performs sparse refinement and selects the top-k anchors and features. Moreover, while existing multi-view attention mechanisms rely on projecting reference points to multiple images, our novel Circular Deformable Attention mechanism only projects to a single image but allows reference points to seamlessly attend to adjacent images, improving efficiency without compromising on performance. On the nuScenes val set, it achieves 46.4% mAP and 55.0% NDS with a ResNet101 backbone. MDHA significantly outperforms the baseline where anchor proposals are modelled as learnable embeddings. Code is available at https://github.com/NaomiEX/MDHA.
BibTeX
@inproceedings{adeline2024mdha,
  author = {Michelle Adeline and Junn Yong Loo and Vishnu Monn Baskaran},
  title = {{MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=gLjpazMUrf},
}

Loading related content...

Figure