DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

Jiasen Wang, Zhenglin Li, Ke Sun, Xianyuan Liu, Yang Zhou, Jian Luan, Bin Wang, Yong Liu

Abstract

Vision-based 3D object detection for autonomous driving typically relies on multiple cameras covering a 360° field of view. Current sparse query-based methods face challenges due to redundant global attention and geometric complexity in mapping 3D coordinates to varying camera poses. We propose Divided View Position Embedding (DVPE), a novel framework that partitions the 3D world into local virtual spaces to decouple position embeddings from camera poses. By combining this approach with visibility-aware cross-attention and enhanced temporal modeling using 2D visual priors, DVPE significantly reduces interference and improves convergence. Experimental results on the nuScenes benchmark demonstrate state-of-the-art performance, achieving 57.2% mAP and 64.5% NDS. Ablation studies further validate the importance of localized geometric reasoning in achieving high detection accuracy and training stability.

Introduction

Autonomous driving systems rely on multi-camera setups to provide a comprehensive 360° view of the environment. While sparse query-based methods have become a standard for 3D object detection, they often struggle with two primary issues. First, redundant global attention allows every query to interact with image features from all cameras, introducing significant noise. Second, encoding 3D coordinates in a global ego-coordinate system forces the model to learn complex, non-linear relationships between positions and camera poses, hindering convergence.

Methodology

DVPE addresses these limitations through three core innovations designed to simplify the geometric learning task and improve feature focus.

Divided View Position Embedding (DVPE)

Instead of utilizing a single global 3D space, DVPE partitions the environment into several frustum-shaped local virtual spaces based on the cameras' fields of view. By transforming 3D reference points into these localized spaces, the model decouples position embeddings from specific camera configurations. This transformation simplifies the mapping process, allowing the transformer to learn more efficiently.

The DVPE framework partitions the 3D environment into local virtual spaces to simplify geometric reasoning across multiple camera views.

Visibility Cross-Attention

To mitigate the noise associated with global attention, we implement visibility-aware cross-attention. Within each virtual space, queries are restricted to interacting only with image features visible to that specific space. This localized attention mechanism acts as a powerful inductive bias, filtering out irrelevant visual information from unrelated views while maintaining a high-fidelity receptive field for target objects.

Visibility cross-attention restricts query interaction to relevant camera views, reducing interference and noise.

Object-Centric Temporal Modeling

DVPE further enhances detection accuracy by incorporating historical 2D Region of Interest (RoI) features. By leveraging mature 2D detection priors, the model can fuse high-level semantic information with 3D features. These priors are cached in a temporal memory queue, ensuring consistency across frames and simplifying the transition from 2D image patches to 3D object representations.

Experimental Evaluation

We evaluated the DVPE framework on the nuScenes dataset, a large-scale benchmark for autonomous driving featuring 1,000 scenes with full 360° camera coverage.

Benchmark Results

DVPE achieved state-of-the-art performance on the nuScenes test set, reaching 57.2% mean Average Precision (mAP) and a 64.5% nuScenes Detection Score (NDS). These results represent a significant improvement over previous leading architectures, such as Sparse4Dv2.

Ablation Analysis

The effectiveness of the divided view approach was validated through ablation experiments. Removing the localized partitioning led to a performance decrease of 2.7% mAP and 3.6% NDS, confirming that the decoupling of geometry and camera poses is the primary driver of the model's success.

Conclusion

Divided View Position Embedding provides a robust solution to the challenges of multi-camera 3D object detection. By localizing geometric reasoning and utilizing visibility-aware attention, DVPE reduces noise and improves training stability. Future work will explore optimizing the computational overhead of space partitioning and further refining the integration of temporal visual priors.

Summary updated

BibTeX
@inproceedings{wang2024dvpe,
  author = {Jiasen Wang and Zhenglin Li and Ke Sun and Xianyuan Liu and Yang Zhou and Jian Luan and Bin Wang and Yong Liu},
  title = {{DVPE: Divided View Position Embedding for Multi-View 3D Object Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure