Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Adrian Lopez Rodriguez, Benjamin Busam, Krystian Mikolajczyk

Abstract

Depth completion is essential for autonomous systems but faces a significant data bottleneck due to the difficulty of obtaining accurate real-world ground truth. While synthetic data provides perfect labels, models trained on it often fail in real-world scenarios due to the domain gap caused by sensor artifacts. This paper introduces "Project to Adapt," a framework that explicitly models geometric noise and employs a novel filtering technique to leverage unlabeled real-world data. Our method improves state-of-the-art performance on the KITTI benchmark by over 6% without requiring expensive real-world annotations.

Introduction: The Data Bottleneck in Depth Completion

Depth completion involves predicting a high-resolution depth map from sparse LiDAR measurements and dense RGB images. A primary challenge in this field is the lack of high-quality real-world training data. Existing datasets like KITTI are often noisy and prone to errors. Synthetic simulators like CARLA offer a solution by providing perfect ground truth, but models trained on "clean" synthetic data struggle with real-world sensor artifacts.

Figure 2: Comparison of synthetic and real domains, highlighting the "see-through" artifacts caused by sensor misalignment.

A critical component of this domain gap is the "see-through" artifact. Because LiDAR and cameras are physically offset, projecting LiDAR points onto the camera's frame causes background points to appear as if they belong to foreground objects, such as tree trunks or cyclists.

Methodology

The "Project to Adapt" framework addresses the domain gap through three main components: geometric sensor mimicking, RGB style adaptation, and reliable point filtering.

Figure 3: The overall "Project to Adapt" pipeline, showing the transition from simulator data to real-world adaptation and the filtering of reliable points.

1. Geometric Sensor Mimicking

Instead of simple random sparsification, the authors simulate the physical multicamera-LiDAR setup within the CARLA simulator. By reprojecting depth from a virtual LiDAR position to the camera's reference frame, they intentionally introduce the same "see-through" artifacts found in real-world data.

2. RGB Style Adaptation

To handle visual differences, a CycleGAN is employed to translate synthetic images into the visual style of the target real-world domain, adjusting for lighting, texture, and color.

3. Reliable Point Filtering ($S_p$)

To utilize unlabeled real-world data, the framework identifies "reliable points" from noisy raw LiDAR input. By applying a minimum pooling strategy within local windows, the system assumes that the closest points are most likely part of an occluding foreground object and thus more reliable. These points serve as a sparse supervisory signal during the second stage of training.

Results and Key Insights

The proposed method was evaluated on the KITTI Depth Completion Benchmark and PandaSet. Key findings include:

  • Noise Modeling is Paramount: Modeling the geometric distribution and noise of the LiDAR is more critical for performance than visual style transfer.
  • State-of-the-Art Performance: The framework improved the state-of-the-art on KITTI by 6.4% in RMSE and 6.3% in MAE among methods not using real-world ground truth.
  • Robustness: The use of Mean Absolute Error (MAE) loss proved more robust than Reverse Huber loss when training with the remaining outliers in the "reliable" point set.

Conclusion and Future Work

"Project to Adapt" demonstrates that explicitly modeling sensor artifacts and filtering for reliable real-world signals can effectively bridge the synthetic-to-real gap in depth completion. Future work will focus on addressing other noise types, such as missing points from low-reflectance surfaces, and developing more computationally efficient, end-to-end training architectures.

Summary updated

BibTeX
@article{rodriguez2023project,
  author = {Adrian Lopez Rodriguez and Benjamin Busam and Krystian Mikolajczyk},
  title = {{Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data}},
  journal = {Int. J. Comput. Vis.},
  year = {2023},
  doi = {10.1007/s11263-022-01726-1},
}
Figure