DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

Jiasen Wang, Zhenglin Li, Ke Sun, Xianyuan Liu, Yang Zhou

Abstract

Multi-view 3D object detection is a critical task for autonomous driving, yet current sparse query-based paradigms suffer from redundant global attention and complex coordinate mapping. We present Divided View Position Embedding (DVPE), a framework that partitions the 3D world into local virtual spaces. By decoupling spatial learning from camera poses and implementing visibility-aware cross-attention, DVPE reduces interference and achieves state-of-the-art performance on the nuScenes benchmark, reaching 57.2% mAP and 64.5% NDS.

Introduction

In autonomous driving, reconstructing a 3D scene from a set of 2D surround-view cameras is a fundamental challenge. Modern architectures often employ sparse query-based methods, where 3D object queries interact with multi-view image features. However, these methods face two primary bottlenecks. First, redundant attention occurs when global attention mechanisms force every 3D query to look at all pixels across all cameras, introducing significant noise from irrelevant views. Second, coordinate complexity arises because the model must learn to map 3D world coordinates back to 2D image planes across varying camera poses, hindering convergence.

Global attention mechanisms often lead to redundant noise as queries attend to irrelevant camera views.

Methodology

The DVPE framework introduces a "divide and conquer" strategy to address these challenges through three core components.

Divided View Position Embedding (DVPE)

Instead of a monolithic global coordinate system, DVPE partitions the 3D space into multiple frustum-shaped local virtual spaces aligned with each camera's field of view. By transforming 3D reference points into these localized coordinates, the model decouples position embeddings from specific extrinsic camera parameters. This geometric simplification allows the transformer to learn spatial relationships more efficiently.

Visibility Cross-Attention

Leveraging the divided virtual spaces, the model implements a visibility mechanism. Queries are restricted to performing cross-attention only with image features that fall within their corresponding virtual space. This targeted focus mimics human perception and drastically reduces computational interference from non-overlapping views.

The proposed visibility-aware mechanism restricts attention to relevant camera frustums, minimizing interference.

Enhanced Temporal and Supervision Strategies

To improve object tracking, DVPE incorporates 2D historical RoI features from a 2D detector, providing high-level semantic anchors for temporal fusion. Additionally, a one-to-many assignment strategy is used during training, providing denser supervision signals to the decoder and stabilizing the learning process.

Results and Discussion

DVPE was evaluated on the nuScenes dataset, the industry standard for multi-view 3D detection. The model achieved state-of-the-art results with 57.2% mAP and 64.5% NDS on the test set. Ablation studies highlight the efficacy of the divided view approach, which alone contributed a 2.4% mAP improvement over the StreamPETR baseline.

The primary insight of this work is that spatial localization—restricting the model's field of regard—paradoxically enhances its intelligence by filtering out environmental noise. By simplifying the underlying trigonometry through local virtual spaces, DVPE allows the network to focus on the semantic task of object detection rather than geometric reconstruction.

Conclusion

DVPE demonstrates that partitioning 3D space into camera-specific virtual volumes significantly improves the accuracy and efficiency of multi-view 3D object detection. By addressing the noise inherent in global attention and the complexity of camera-to-world mapping, DVPE sets a new standard for sparse query-based perception in autonomous systems.

Summary updated

BibTeX
@inproceedings{wang2024dvpe,
  author = {Jiasen Wang and Zhenglin Li and Ke Sun and Xianyuan Liu and Yang Zhou},
  title = {{DVPE: Divided View Position Embedding for Multi-View 3D Object Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure