Abstract
Vision-based 3D object detection for autonomous driving typically relies on multiple cameras covering a 360° field of view. Current sparse query-based methods face challenges due to redundant global attention and geometric complexity in mapping 3D coordinates to varying camera poses. We propose Divided View Position Embedding (DVPE), a novel framework that partitions the 3D world into local virtual spaces to decouple position embeddings from camera poses. By combining this approach with visibility-aware cross-attention and enhanced temporal modeling using 2D visual priors, DVPE significantly reduces interference and improves convergence. Experimental results on the nuScenes benchmark demonstrate state-of-the-art performance, achieving 57.2% mAP and 64.5% NDS. Ablation studies further validate the importance of localized geometric reasoning in achieving high detection accuracy and training stability.
Introduction
Autonomous driving systems rely on multi-camera setups to provide a comprehensive 360° view of the environment. While sparse query-based methods have become a standard for 3D object detection, they often struggle with two primary issues. First, redundant global attention allows every query to interact with image features from all cameras, introducing significant noise. Second, encoding 3D coordinates in a global ego-coordinate system forces the model to learn complex, non-linear relationships between positions and camera poses, hindering convergence.
Methodology
DVPE addresses these limitations through three core innovations designed to simplify the geometric learning task and improve feature focus.
Divided View Position Embedding (DVPE)
Instead of utilizing a single global 3D space, DVPE partitions the environment into several frustum-shaped local virtual spaces based on the cameras' fields of view. By transforming 3D reference points into these localized spaces, the model decouples position embeddings from specific camera configurations. This transformation simplifies the mapping process, allowing the transformer to learn more efficiently.
Visibility Cross-Attention
To mitigate the noise associated with global attention, we implement visibility-aware cross-attention. Within each virtual space, queries are restricted to interacting only with image features visible to that specific space. This localized attention mechanism acts as a powerful inductive bias, filtering out irrelevant visual information from unrelated views while maintaining a high-fidelity receptive field for target objects.
Object-Centric Temporal Modeling
DVPE further enhances detection accuracy by incorporating historical 2D Region of Interest (RoI) features. By leveraging mature 2D detection priors, the model can fuse high-level semantic information with 3D features. These priors are cached in a temporal memory queue, ensuring consistency across frames and simplifying the transition from 2D image patches to 3D object representations.
Experimental Evaluation
We evaluated the DVPE framework on the nuScenes dataset, a large-scale benchmark for autonomous driving featuring 1,000 scenes with full 360° camera coverage.
Benchmark Results
DVPE achieved state-of-the-art performance on the nuScenes test set, reaching 57.2% mean Average Precision (mAP) and a 64.5% nuScenes Detection Score (NDS). These results represent a significant improvement over previous leading architectures, such as Sparse4Dv2.
Ablation Analysis
The effectiveness of the divided view approach was validated through ablation experiments. Removing the localized partitioning led to a performance decrease of 2.7% mAP and 3.6% NDS, confirming that the decoupling of geometry and camera poses is the primary driver of the model's success.
Conclusion
Divided View Position Embedding provides a robust solution to the challenges of multi-camera 3D object detection. By localizing geometric reasoning and utilizing visibility-aware attention, DVPE reduces noise and improves training stability. Future work will explore optimizing the computational overhead of space partitioning and further refining the integration of temporal visual priors.
Summary updated