Back to the Color: Learning Depth to Specific Color Transformation for Unsupervised Depth Estimation

Yufan Zhu, Chongzhi Ran, Mingtao Feng, Weisheng Dong, Antonio M. López, Guangming ShiarXiv 2024 · arXiv · OpenReview

Abstract

Virtual engines can generate dense depth maps for various synthetic scenes, making them invaluable for training depth estimation models. However, discrepancies between synthetic and real-world colors pose significant challenges for depth estimation in real-world scenes, especially in complex and uncertain environments encountered in unsupervised monocular depth estimation tasks. To address this issue, we propose Back2Color, a framework that predicts realistic colors from depth using a model trained on real-world data, thus transforming synthetic colors into their real-world counterparts. Additionally, we introduce the Syn-Real CutMix method for joint training with both real-world unsupervised and synthetic supervised depth samples, enhancing monocular depth estimation performance in real-world scenes. Furthermore, to mitigate the impact of non-rigid motions on depth estimation, we present an auto-learning uncertainty temporal-spatial fusion method (Auto-UTSF), which leverages the strengths of unsupervised learning in both temporal and spatial dimensions. We also designed VADepth, based on the Vision Attention Network, which offers lower computational complexity and higher accuracy than transformers. Our Back2Color framework achieves state-of-the-art performance on the Kitti dataset, as evidenced by improvements in performance metrics and the production of fine-grained details.

Introduction

Unsupervised monocular depth estimation—predicting 3D distance from a single 2D image—is a cornerstone of autonomous driving research. While attractive due to the lack of manual labeling requirements, these methods face two primary hurdles: the domain gap and dynamic objects. Synthetic data provides perfect ground-truth depth but lacks realism, while real-world data is realistic but lacks dense labels. Furthermore, non-rigid motions (like moving cars) often confuse models, leading to artifacts like "depth clouds."

The Back2Color framework addresses these challenges by bridging the synthetic-to-real gap. It learns to "paint" realistic real-world textures onto synthetic depth maps and employs an uncertainty-aware fusion of temporal and spatial cues to handle dynamic scenes.

Methodology

The Back2Color framework introduces two primary innovations: a depth-to-color transformation and an uncertainty-aware fusion module.

1. Back2Color: Depth-to-Color Transformation

Instead of traditional image-to-image translation, which can distort geometry, Back2Color uses a bidirectional approach. A "ColorNet" is trained to predict real-world colors from estimated depth maps. Once trained, synthetic depth maps are fed into this network to generate "Syn2Real" images. These images retain the perfect geometry of the simulation while adopting the color distribution and style of the real world, creating a high-quality paired dataset for supervised training.

The two-step optimization process: Step I trains the depth-to-color model on real data, and Step II uses it to transform synthetic data for supervised training.

2. Auto-UTSF: Uncertainty Temporal-Spatial Fusion

To handle moving objects, the framework fuses temporal supervision (comparing video frames) with spatial supervision (stereo pairs). An "Auto-learning Uncertainty" module predicts pixel-wise confidence. When temporal logic fails due to motion, the model automatically shifts its trust to spatial stereo cues, ensuring stability in complex traffic.

3. Backbone and Data

The authors utilize the Visual Attention Network (VAN), a lightweight CNN-based backbone that provides a balance of efficiency and accuracy. Training involves a mix of real-world data (KITTI, Cityscapes) and a unified synthetic dataset ("MixSyn") derived from vKITTI2, SHIFT, and UrbanSyn.

Results and Discussion

The Back2Color framework achieves state-of-the-art performance on the KITTI benchmark, reaching an AbsRel of 0.093, outperforming previous leaders like MonoViT (0.096).

Efficiency vs. Performance

A key advantage of the proposed VADepth (VAN-based) models is their computational efficiency. The "VAN0" variant uses only 5.17M parameters and 17.64 GFLOPs, making it significantly leaner than transformer-based alternatives like MonoViT, which requires over 33M parameters and 159 GFLOPs.

Performance (a1 metric) vs. Model Complexity (Parameters and GFLOPs). Back2Color variants (red triangles) consistently outperform other models at lower computational costs.

Qualitative Improvements

Visual results demonstrate sharper object boundaries and reduced artifacts around moving pedestrians and vehicles. The model effectively leverages geometry as an anchor, forcing it to learn structural features that generalize across domains.

Conclusion

Back2Color demonstrates that mapping color to geometry and back again allows for effective domain adaptation without geometric distortion. While the depth-to-color network occasionally "hallucinates" non-geometric details like lane markings, this behavior actually aids in bridging the domain gap. Future work will explore the application of "Syn2Real" images to other tasks like semantic segmentation and testing under extreme weather conditions.

Summary updated

Abstract

Virtual engines can generate dense depth maps for various synthetic scenes, making them invaluable for training depth estimation models. However, discrepancies between synthetic and real-world colors pose significant challenges for depth estimation in real-world scenes, especially in complex and uncertain environments encountered in unsupervised monocular depth estimation tasks. To address this issue, we propose Back2Color, a framework that predicts realistic colors from depth using a model trained on real-world data, thus transforming synthetic colors into their real-world counterparts. Additionally, we introduce the Syn-Real CutMix method for joint training with both real-world unsupervised and synthetic supervised depth samples, enhancing monocular depth estimation performance in real-world scenes. Furthermore, to mitigate the impact of non-rigid motions on depth estimation, we present an auto-learning uncertainty temporal-spatial fusion method (Auto-UTSF), which leverages the strengths of unsupervised learning in both temporal and spatial dimensions. We also designed VADepth, based on the Vision Attention Network, which offers lower computational complexity and higher accuracy than transformers. Our Back2Color framework achieves state-of-the-art performance on the Kitti dataset, as evidenced by improvements in performance metrics and the production of fine-grained details. This is particularly evident on more challenging datasets such as Cityscapes for unsupervised depth estimation.
BibTeX
@misc{zhu2024back,
  author = {Yufan Zhu and Chongzhi Ran and Mingtao Feng and Weisheng Dong and Antonio M. López and Guangming Shi},
  title = {{Back to the Color: Learning Depth to Specific Color Transformation for Unsupervised Depth Estimation}},
  year = {2024},
  eprint = {2406.07741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07741},
}
Figure