Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GAN

Alona Baruhov, Guy GilboaarXiv 2020 · arXiv

Introduction

Depth sensing is a cornerstone of modern robotics and augmented reality. However, a significant gap exists between high-end sensors like the Microsoft Kinect 2 and low-cost alternatives like the Intel RealSense. Low-cost sensors often produce "Low-Quality" (LQ) images characterized by structural artifacts, non-Gaussian noise, and "black holes"—regions where depth information is missing due to surface reflectivity or sensor limitations. Enhancing these images typically requires paired datasets for supervised learning, which are difficult to acquire and align in real-world settings. This paper presents Tri-Cycle GAN, an unsupervised framework that bridges this gap without requiring paired ground-truth data.

Methodology

The authors identify a fundamental flaw in applying standard Cycle-GAN to depth enhancement: domain asymmetry. While a "High-Quality" (HQ) image can be degraded into many possible LQ versions (one-to-many), the reverse is not true. Standard cycle-consistency ($F(G(x)) = x$) forces the network to preserve noise patterns to satisfy the identity requirement, which mathematically hinders actual enhancement.

The Tri-Cycle Loss

To address asymmetry, the authors introduce the Tri-Cycle loss. Instead of a simple round-trip ($L \to H \to L$), they propose a three-step cycle: $L \to H \to L \to H$. The constraint is that the second HQ output must match the first: $G(F(G(Z_L))) = G(Z_L)$. This allows the degrader ($F$) to be one-to-many while ensuring the enhancer ($G$) remains consistent and focused on producing stable, clean depth maps.

Qualitative comparison of depth map enhancement showing the transition from noisy LQ input to enhanced HQ output.

Architectural Improvements

  1. U-Net Generator: A deeper and wider U-Net architecture replaces the standard generator, providing the capacity to fill large "black holes" and resolve complex scene geometry.
  2. Masked Similarity Loss: To handle invalid pixels (zeros) common in depth maps, the authors implement masked losses. This ensures the network only calculates loss on valid data regions, preventing it from trying to reconstruct random noise patterns found in the HQ domain.

Experimental Evaluation

The framework was validated using three distinct datasets:

  • Synthetic Dataset: 120,000 images with simulated real-world noise (structural noise and adaptive holes).
  • Office Dataset: A truly unsupervised set of ~1,000 unpaired images from RealSense R200 (LQ) and Kinect 2 (HQ).
  • DROT Dataset: A small, publicly available dataset of 112 nearly-aligned image sets used for quantitative benchmarking.

Results

Tri-Cycle GAN demonstrates superior performance over both vanilla Cycle-GAN and non-guided methods like Sparse Depth Sensing (SDS). Using Patch-based Normalized Cross-Correlation (PNCC), the method achieved a score of 0.879 on synthetic data (vs. 0.668 for base Cycle-GAN) and 0.633 on the DROT dataset (vs. 0.213 for base Cycle-GAN).

A high-quality depth map output demonstrating sharp edges and clean surface reconstruction.

Qualitatively, the model produces sharper object boundaries and more realistic hole-filling. By relaxing the strict cycle-consistency, the network successfully ignores the specific noise of the LQ sensor while adopting the superior geometric representation of the HQ domain.

Conclusion

Tri-Cycle GAN provides a robust, unsupervised solution for enhancing depth images from low-cost sensors. By reformulating cycle-consistency to accommodate domain asymmetry, the method achieves state-of-the-art results on real-world data. Future work will explore applying this "Tri-Cycle" logic to other asymmetric translation tasks, such as super-resolution or image dehazing, where the degradation process is inherently lossy.

Summary updated

Abstract

Low quality depth poses a considerable challenge to computer vision algorithms. In this work we aim to enhance highly degraded, real-world depth images acquired by a low-cost sensor, for which an analytical noise model is unavailable. In the absence of clean ground-truth, we approach the task as an unsupervised domain-translation between the low-quality sensor domain and a high-quality sensor domain, represented using two unpaired training sets. We employ the highly-successful Cycle-GAN to this task, but find it to perform poorly in this case. Identifying the sources of the failure, we introduce several modifications to the framework, including a larger generator architecture, depth-specific losses that take into account missing pixels, and a novel Tri-Cycle loss which promotes information-preservation while addressing the asymmetry between the domains. We show that the resulting framework dramatically improves over the original Cycle-GAN both visually and quantitatively, extending its applicability to more challenging and asymmetric translation tasks.
BibTeX
@misc{baruhov2020unsupervised,
  author = {Alona Baruhov and Guy Gilboa},
  title = {{Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GAN}},
  year = {2020},
  eprint = {2001.03779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.03779},
}
Figure