Towards Domain Generalization for Multi-view 3D Object Detection in Bird-Eye-View

Shuo Wang, Xinhai Zhao, Hai-Ming Xu, Zehui Chen, Dameng Yu, Jiahao Chang, Zhen Yang, Feng Zhao

Abstract

Multi-view 3D object detection (MV3D-Det) in Bird-Eye-View (BEV) is a cost-effective alternative to LiDAR-based systems but suffers from significant performance degradation when deployed across different camera configurations. This "domain gap" is primarily driven by depth-shift errors caused by variations in camera intrinsics and extrinsics. We present DG-BEV, a framework that addresses these challenges through intrinsics-decoupled depth prediction, dynamic perspective augmentation, and adversarial domain-invariant feature learning. Experimental results across nuScenes, Waymo, and Lyft datasets demonstrate that DG-BEV achieves up to a 7x increase in detection accuracy on unseen domains without sacrificing source domain performance.

1. Introduction

Multi-view 3D object detection (MV3D-Det) has emerged as a critical component for autonomous driving systems seeking to minimize hardware costs. Despite its potential, MV3D-Det models are notoriously sensitive to variations in camera parameters, including focal lengths, mounting heights, and angles. When a model trained on a specific fleet is deployed on a vehicle with a different sensor setup, performance typically collapses. This degradation is largely attributed to a shift in the BEV feature distribution, specifically a "depth-shift" phenomenon where objects are incorrectly localized along the depth axis due to the model's reliance on specific training camera geometries.

Qualitative comparison showing how the baseline (left) suffers from depth-shift (green arrows) on a new domain, while DG-BEV (right) correctly localizes objects.

2. Methodology

The DG-BEV framework addresses the domain gap through three integrated strategies designed to make the detection pipeline domain-agnostic.

The overall DG-BEV framework, highlighting the depth decoupling (i), perspective augmentation (ii), and adversarial feature learning (iii).

2.1 Intrinsics-Decoupled Depth Prediction (IDD)

Standard depth estimation modules often learn to associate specific image scales with metric depths, effectively "hard-coding" the training camera's focal length into the neural network weights. DG-BEV instead predicts a scale-invariant depth and uses the camera's known focal length as a mathematical bridge to recover metric depth. This prevents the model from miscalculating distance when the focal length changes.

2.2 Dynamic Perspective Augmentation (DPA)

To ensure robustness against varying extrinsic parameters, such as camera mounting heights and angles, the framework employs homography-based warping. This simulates diverse camera orientations (pitch, yaw, and roll) during training, forcing the model to learn the underlying geometry of the scene rather than memorizing a fixed viewpoint.

2.3 Domain-Invariant Feature Learning (DIFL)

DG-BEV creates "pseudo-domains" by artificially scaling images to simulate different focal lengths. An adversarial setup using a Gradient Reverse Layer (GRL) is then employed to "punish" the model if it can distinguish between these domains. This encourages the backbone to extract features that are useful for detection but invariant to camera-specific quirks. The researchers found that using an Ordinal Loss for these pseudo-domains performed best, as focal length is a continuous physical property.

3. Experimental Results

The framework was evaluated using nuScenes as the primary source domain, with the Waymo Open Dataset and Lyft L5 Dataset as target domains to test cross-dataset generalization.

3.1 Cross-Dataset Generalization

The results demonstrate a dramatic improvement in zero-shot generalization. For the nuScenes to Waymo transition, the baseline model achieved a Mean Average Precision (mAP) of only 0.040. DG-BEV improved this to 0.297—a massive gain that makes the model viable in the new domain. Similarly, for Waymo to nuScenes, mAP increased from 0.032 to 0.303.

3.2 Source Domain Stability

Crucially, the improvements in generalization did not come at the cost of performance on the original training data. The model maintained its original accuracy levels on the source domain, indicating that the domain-agnostic features are equally effective for the training distribution.

4. Conclusion

DG-BEV successfully mitigates the geometric domain gap in MV3D-Det by decoupling depth estimation and enforcing feature invariance. While this work focuses on geometric projections, future research should address the semantic gap—differences in environmental conditions, lighting, and regional object appearances—to develop a truly universal 3D detection system.

Summary updated

Abstract

Multi-view 3D object detection (MV3D-Det) in Bird-Eye-View (BEV) has drawn extensive attention due to its low cost and high efficiency. Although new algorithms for camera-only 3D object detection have been continuously proposed, most of them may risk drastic performance degradation when the domain of input images differs from that of training. In this paper, we first analyze the causes of the domain gap for the MV3D-Det task. Based on the covariate shift assumption, we find that the gap mainly attributes to the feature distribution of BEV, which is determined by the quality of both depth estimation and 2D image's feature representation. To acquire a robust depth prediction, we propose to decouple the depth estimation from the intrinsic parameters of the camera (i.e. the focal length) through converting the prediction of metric depth to that of scale-invariant depth and perform dynamic perspective augmentation to increase the diversity of the extrinsic parameters (i.e. the camera poses) by utilizing homography. Moreover, we modify the focal length values to create multiple pseudo-domains and construct an adversarial training loss to encourage the feature representation to be more domain-agnostic. Without bells and whistles, our approach, namely DG-BEV, successfully alleviates the performance drop on the unseen target domain without impairing the accuracy of the source domain. Extensive experiments on various public datasets, including Waymo, nuScenes, and Lyft, demonstrate the generalization and effectiveness of our approach. To the best of our knowledge, this is the first systematic study to explore a domain generalization method for MV3D-Det.
BibTeX
@inproceedings{wang2023towards,
  author = {Shuo Wang and Xinhai Zhao and Hai-Ming Xu and Zehui Chen and Dameng Yu and Jiahao Chang and Zhen Yang and Feng Zhao},
  title = {{Towards Domain Generalization for Multi-view 3D Object Detection in Bird-Eye-View}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure