Abstract
Depth completion is essential for autonomous systems but faces a significant data bottleneck due to the difficulty of obtaining accurate real-world ground truth. While synthetic data provides perfect labels, models trained on it often fail in real-world scenarios due to the domain gap caused by sensor artifacts. This paper introduces "Project to Adapt," a framework that explicitly models geometric noise and employs a novel filtering technique to leverage unlabeled real-world data. Our method improves state-of-the-art performance on the KITTI benchmark by over 6% without requiring expensive real-world annotations.
Introduction: The Data Bottleneck in Depth Completion
Depth completion involves predicting a high-resolution depth map from sparse LiDAR measurements and dense RGB images. A primary challenge in this field is the lack of high-quality real-world training data. Existing datasets like KITTI are often noisy and prone to errors. Synthetic simulators like CARLA offer a solution by providing perfect ground truth, but models trained on "clean" synthetic data struggle with real-world sensor artifacts.
A critical component of this domain gap is the "see-through" artifact. Because LiDAR and cameras are physically offset, projecting LiDAR points onto the camera's frame causes background points to appear as if they belong to foreground objects, such as tree trunks or cyclists.
Methodology
The "Project to Adapt" framework addresses the domain gap through three main components: geometric sensor mimicking, RGB style adaptation, and reliable point filtering.
1. Geometric Sensor Mimicking
Instead of simple random sparsification, the authors simulate the physical multicamera-LiDAR setup within the CARLA simulator. By reprojecting depth from a virtual LiDAR position to the camera's reference frame, they intentionally introduce the same "see-through" artifacts found in real-world data.
2. RGB Style Adaptation
To handle visual differences, a CycleGAN is employed to translate synthetic images into the visual style of the target real-world domain, adjusting for lighting, texture, and color.
3. Reliable Point Filtering ($S_p$)
To utilize unlabeled real-world data, the framework identifies "reliable points" from noisy raw LiDAR input. By applying a minimum pooling strategy within local windows, the system assumes that the closest points are most likely part of an occluding foreground object and thus more reliable. These points serve as a sparse supervisory signal during the second stage of training.
Results and Key Insights
The proposed method was evaluated on the KITTI Depth Completion Benchmark and PandaSet. Key findings include:
- Noise Modeling is Paramount: Modeling the geometric distribution and noise of the LiDAR is more critical for performance than visual style transfer.
- State-of-the-Art Performance: The framework improved the state-of-the-art on KITTI by 6.4% in RMSE and 6.3% in MAE among methods not using real-world ground truth.
- Robustness: The use of Mean Absolute Error (MAE) loss proved more robust than Reverse Huber loss when training with the remaining outliers in the "reliable" point set.
Conclusion and Future Work
"Project to Adapt" demonstrates that explicitly modeling sensor artifacts and filtering for reliable real-world signals can effectively bridge the synthetic-to-real gap in depth completion. Future work will focus on addressing other noise types, such as missing points from low-reflectance surfaces, and developing more computationally efficient, end-to-end training architectures.
Summary updated