Abstract
Transformer-based approaches advance the recent development of multi-camera 3D detection both in academia and industry. In a vanilla transformer architecture, queries are randomly initialised and optimised for the whole dataset, without considering the differences among input frames. In this work, we propose to leverage the predictions from an image backbone, which is often highly optimised for 2D tasks, as priors to the transformer part of a 3D detection network. The method works by (1) augmenting image feature maps with 2D priors, (2) sampling query locations via ray-casting along 2D box centroids, as well as (3) initialising query features with object-level image features. Experimental results show that 2D priors not only help the model converge faster, but also largely improve the baseline approach by up to 12% in terms of average precision.
Introduction
Multi-camera 3D object detection is a cornerstone of autonomous driving systems. Recent state-of-the-art models, such as DETR3D, utilize transformer decoders to query image features and predict 3D bounding boxes. However, these "vanilla" architectures typically initialize queries randomly, requiring the model to learn spatial relationships from scratch across the entire dataset. This approach suffers from two primary issues: slow training convergence and the "smearing" effect, where depth ambiguity leads to false positives along a camera's line of sight. This paper proposes to solve these issues by leveraging highly optimized 2D neural networks to "prime" the 3D transformer.
Proposed Methodology
The authors propose a framework that incorporates 2D image-level knowledge into the 3D detection pipeline through three main strategies:
1. Feature Map Priors
Standard image feature maps are augmented by concatenating them with 2D semantic segmentation and depth maps. This provides the transformer’s cross-attention mechanism with explicit geometric and semantic hints, clarifying object boundaries and relative distances before the 3D reasoning begins.
2. Location and Query Priors
A significant innovation in this work is the transition from random 3D "anchor" points to informed location priors. The system identifies 2D bounding box centroids and performs ray-casting, sampling 3D points at 5-meter intervals along the ray extending from the camera through the 2D center. This effectively narrows the transformer's search space from the entire 3D volume to specific lines of sight.
Additionally, query features are no longer initialized with random embeddings. Instead, the model crops features from the 2D box area, weights them by semantic scores, and processes them through a Multi-Layer Perceptron (MLP) to create a starting "description" of the object. This initialization allows the transformer to focus on refining the object's properties rather than discovering them from scratch.
Experimental Results
The method was evaluated on a large-scale dataset comprising approximately 60k training frames and 10k validation samples. The integration of feature, location, and query priors yielded substantial performance gains:
- Accuracy Boost: The full model improved the DETR3D baseline by 11.44% AP for vehicles and 8.32% AP for humans.
- Convergence Speed: The model reached target error rates in significantly fewer epochs compared to the vanilla baseline, demonstrating that priors act as a massive headstart for training.
- Robustness: For human detection, using a more relaxed distance metric (4m) resulted in a >30% improvement, highlighting the effectiveness of priors for small-scale object localization where depth estimation is traditionally difficult.
Discussion and Conclusion
The success of this approach stems from the fact that 2D vision is largely a "solved" shortcut. Convolutional neural networks (CNNs) are exceptionally efficient at extracting semantic information from 2D images. By solving the "where to look" problem via 2D detections, the transformer is left to solve the much simpler "how far away" problem. This targeted approach also mitigates the "smearing" effect, as the model is less likely to hallucinate objects in empty space.
In conclusion, 2D priors are a powerful tool for enhancing 3D transformer-based detectors. Future work will explore the inclusion of temporal priors like scene flow and the extension of this framework to multi-modal sensor fusion involving Lidar and Radar.
Summary updated