Introduction
3D object detection is a cornerstone of autonomous driving, yet models often experience significant performance degradation when deployed in environments different from their training data—a challenge known as the cross-domain problem. This paper investigates the root causes of this failure, revealing that current models tend to overfit to the specific object dimensions of the source dataset rather than learning generalizable geometric features.
The Cross-Domain Challenge
For autonomous systems to scale, they must operate reliably across various cities, weather conditions, and sensor configurations. However, a model trained on one dataset (source domain) typically fails when tested on another (target domain). While domain adaptation techniques exist, the authors suspect they often act as a "band-aid," masking structural flaws in how models perceive spatial data. They set out to analyze whether adding camera data helps or hurts and whether current evaluation metrics are capturing the root of the problem.
Methodology and New Metrics
The authors evaluate several 3D detection architectures, including point-based (PV-RCNN), voxel-based (SECOND-IoU), and transformer-based (TransFusion) models. To better diagnose spatial failures, they propose two novel metrics beyond standard 3D or Bird's-Eye View (BEV) Average Precision (AP):
- Side-view AP: Evaluates length and height accuracy by projecting boxes onto the X-Z plane.
- Front-view AP: Evaluates width and height accuracy by projecting boxes onto the Y-Z plane.
Datasets
The study leverages three major benchmarks, standardizing coordinate systems and voxel sizes to allow for cross-dataset evaluation:
- KITTI: Features 64-beam LiDAR with dense point clouds, focused on front-view annotations.
- nuScenes: Utilizes 32-beam LiDAR, resulting in sparser point clouds.
- Waymo: A large-scale dataset providing high-density point clouds.
Key Insights
The Fragility of Multi-Modal Models
Contrary to intuition, integrating camera data with LiDAR often decreases robustness in cross-domain scenarios. Variations in camera calibration, image resolution, and pixel distributions make multi-modal models more prone to failure than their LiDAR-only counterparts when encountering new domains. For example, a multi-modal TransFusion model collapsed to 0.5% AP on a nuScenes-to-KITTI task, while its LiDAR-only version maintained 17.4%.
The Illusion of Domain Adaptation
Evaluation of the ST3D domain adaptation method suggests it does not necessarily improve fundamental generalization. Instead, it often causes the model to "forget" the source domain and overfit to the target domain, leading to unstable performance depending on the specific subset of target data used.
The Width vs. Length Paradox
A critical finding is that models struggle significantly more with predicting object width across domains than length. Using the proposed front-view AP, the authors discovered that models often memorize the average car width of the training set. Because Waymo cars are roughly 30% wider than KITTI cars, a model trained on Waymo will incorrectly apply those dimensions to KITTI vehicles. In one test, a model achieved a Length AP of 73.9% but a Width AP of only 12.1%.
Conclusion
The authors conclude that the community must move beyond simple 3D/BEV AP metrics, which can hide dimensional overfitting. Future research should focus on developing architectures that perform true geometric reasoning from point clouds rather than memorizing statistical dimensions, ensuring safety and reliability across diverse real-world environments.
Summary updated