Introduction
Training 3D object detection models on simulated LiDAR data is an attractive prospect due to the availability of perfect, automated labeling. However, models trained in simulation often suffer from a significant performance drop when deployed in real-world environments. This "sim-to-real" gap is primarily driven by three domain shifts: statistical disparities in object size distributions, differences in point cloud density (simulators often produce overly dense data), and limited feature diversity in simulated assets.
Existing Unsupervised Domain Adaptation (UDA) methods typically rely on pseudo-labeling, where a model's own predictions on target data serve as ground truth. In sim-to-real scenarios, these labels are often too noisy to be effective. The Complex-to-Simple (CTS) framework addresses this by introducing a noise-aware adaptation strategy that filters unreliable labels using a novel uncertainty metric.
The CTS Framework
The CTS framework operates in two distinct stages to bridge the domain gap and refine pseudo-label quality.
Stage 1: Enhancing Pseudo-Label Quality
To prevent the propagation of simulation-specific biases, CTS introduces two key components:
- Anchor Head (AH): Instead of allowing the model to refine proposals based on biased source sizes, CTS utilizes globally fixed-size 3D anchors. This decouples detection from specific size priors, making the model more robust to varying vehicle dimensions in the real world.
- RoI Random Scaling (RRS): Source data is augmented by randomly scaling objects, forcing the model to learn features independent of specific object dimensions.
Stage 2: Noise-Aware Mean Teacher Adaptation
The core of the adaptation utilizes a Student-Teacher architecture. The Teacher model generates pseudo-labels, while the Student model learns from them. To manage label noise, CTS introduces Corner-Format Aleatoric Uncertainty (AU). Traditional bounding box uncertainty metrics are difficult to compare because they mix different units (meters for size, radians for rotation). CTS instead predicts the uncertainty of the eight corner points of the box, providing a uniform and mathematically consistent metric for model confidence.
Noise-Aware Sampling Strategies
To ensure the model learns from high-quality data, CTS employs a two-level sampling strategy:
- Frame-level Sampling: A curriculum learning approach that selects "clean" frames with low average uncertainty early in training, gradually introducing more challenging frames as the model matures.
- Object-level Sampling: A soft-sampling strategy that weights the loss of each object by the inverse of its corner uncertainty. This effectively ignores high-uncertainty, noisy labels during the training process.
Experimental Results
The CTS framework was evaluated by transferring knowledge from the CARLA3D source dataset (generated via the CARLA simulator) to real-world target datasets including KITTI, Lyft, and TinySUScape.
Results demonstrate that CTS significantly outperforms existing state-of-the-art UDA methods. For the CARLA3D $\rightarrow$ KITTI task, the $AP_{3D}$ (Moderate) improved from a baseline of 4.06% to 45.28%. Similarly, on the CARLA3D $\rightarrow$ Lyft task, performance jumped from 13.85% to 45.87%. Ablation studies confirmed that the corner-based uncertainty and noise-aware sampling provided a critical ~5% boost over standard Mean Teacher setups, highlighting the importance of intelligent data selection over raw data volume.
Conclusion
The CTS framework provides a robust solution for sim-to-real 3D detection by addressing size bias and label noise. By shifting to a uniform corner-based uncertainty metric and employing curriculum-based sampling, it effectively bridges the gap between perfect simulations and messy real-world data. Future work will focus on expanding the diversity of simulated assets and addressing environmental factors like adverse weather to further close the gap with models trained on labeled real-world data.
Summary updated