Multi-View Attentive Contextualization for Multi-View 3D Object Detection

Xianpeng Liu, Ce Zheng, Ming Qian, Nan Xue, Chen Chen, Zhebin Zhang, Chen Li, Tianfu Wu

Abstract

Multi-camera 3D object detection is a cornerstone of autonomous driving, yet current methods struggle with the "lifting" process from 2D image space to 3D representations. We present Multi-View Attentive Contextualization (MvACon), a plug-and-play module designed to provide dense, scene-level context to 2D features before lifting. By utilizing a novel Patch-to-Cluster Attention (PaCa) mechanism, MvACon achieves global 3D awareness with linear computational complexity, significantly improving detection accuracy, velocity estimation, and orientation stability across major benchmarks.

Introduction: The Context Problem in 3D Lifting

In autonomous driving, transforming 2D image data from multiple cameras into a unified Bird’s Eye View (BEV) is critical. Current state-of-the-art methods typically follow one of two paths, both of which face limitations regarding scene context:

  1. Dense Attention (e.g., PETR): These models attempt to process the entire scene simultaneously. While comprehensive, the quadratic computational cost often forces the use of low-resolution features, sacrificing fine-grained detail.
  2. Sparse Attention (e.g., BEVFormer): These models sample specific points in 2D images to represent 3D locations. However, these points are often "shallow"—they may land on semantically irrelevant areas like road patches or sky, lacking the global confidence required for precise object localization.

Both approaches suffer from a lack of global 3D awareness, missing the broader environmental context necessary to understand an object's position and motion relative to its surroundings.

Methodology: MvACon and Patch-to-Cluster Attention

MvACon acts as an intermediary between the 2D image backbone and the 3D lifting module. Its core innovation is Patch-to-Cluster Attention (PaCa).

MvACon integrates into the standard 3D detection pipeline between the 2D backbone and the feature lifting stage.

Instead of standard $O(N^2)$ self-attention, MvACon first learns to group 2D image tokens into a small set of $M$ semantic clusters. Every pixel in the image then performs attention against these clusters. This approach ensures the module is representationally dense (every pixel receives global context) while remaining computationally sparse (linear complexity). This modular design allows MvACon to be integrated into existing architectures like PETR or BEVFormer with minimal overhead.

Qualitative Insights: Context as a Coordinate System

Visual analysis reveals that MvACon does more than just highlight vehicles; it learns to attend to static background elements such as buildings and infrastructure. These stable environmental features act as a local coordinate system, which significantly aids the model in estimating an object's velocity and orientation.

Comparison of attention mechanisms: MvACon (d) provides more stable and semantically grounded attention points compared to the sparse sampling of BEVFormer (b).

Furthermore, visualizations show that the sampling points used in the lifting process become much more spatiotemporally stable across different network layers and consecutive video frames when MvACon is employed, compared to vanilla BEVFormer baselines.

Experimental Results

The researchers evaluated MvACon on two major industry benchmarks: nuScenes and Waymo-mini.

  • nuScenes Performance: MvACon improved the NuScenes Detection Score (NDS) by 0.8 to 1.7 points across various baselines (PETR, BEVFormer, and DFA3D). The most significant gains were seen in error reduction; for instance, in BEVFormer-small, it reduced orientation error (mAOE) by 0.6 and velocity error (mAVE) by 1.6.
  • Waymo-mini: On the Waymo benchmark, it improved the Longitudinal Error Tolerant mAP (LET-mAPL) by 0.8, demonstrating strong generalization across different sensor configurations.
  • Efficiency: A "lite" version of MvACon, which processes feature scales independently, provides the majority of these performance gains with a negligible increase in GPU memory (~8MB).

Conclusion and Future Work

MvACon demonstrates that providing global context to 2D features before 3D lifting is a highly effective strategy for improving multi-view detection. While the full version increases memory consumption when concatenating multi-scale features, the lite version offers a practical alternative for resource-constrained environments. Future research may explore integrating temporal cross-attention directly into the clustering mechanism to further refine motion estimation in dynamic scenes.

Summary updated

Abstract

We present Multi-View Attentive Contextualization (MvACon), a simple yet effective method for improving 2D-to-3D feature lifting in query-based multi-view 3D (MV3D) object detection. Despite remarkable progress witnessed in the field of query-based MV3D object detection, prior art often suffers from either the lack of exploiting high-resolution 2D features in dense attention-based lifting, due to high computational costs, or from insufficiently dense grounding of 3D queries to multi-scale 2D features in sparse attention-based lifting. Our proposed MvACon hits the two birds with one stone using a representationally dense yet computationally sparse attentive feature contextualization scheme that is agnostic to specific 2D-to-3D feature lifting approaches. In experiments, the proposed MvA-Con is thoroughly tested on the nuScenes benchmark, using both the BEVFormer and its recent 3D deformable attention (DFA3D) variant, as well as the PETR, showing consistent detection performance improvement, especially in enhancing performance in location, orientation, and velocity prediction. It is also tested on the Waymo-mini benchmark using BEVFormer with similar improvement. We qualitatively and quantitatively show that global cluster-based contexts effectively encode dense scene-level contexts for MV3D object detection. The promising results of our proposed MvA-Con reinforces the adage in computer vision - “(contextualized) feature matters”.
BibTeX
@inproceedings{liu2024multi,
  author = {Xianpeng Liu and Ce Zheng and Ming Qian and Nan Xue and Chen Chen and Zhebin Zhang and Chen Li and Tianfu Wu},
  title = {{Multi-View Attentive Contextualization for Multi-View 3D Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure