Introduction
Depth sensing is a cornerstone of modern robotics and augmented reality. However, a significant gap exists between high-end sensors like the Microsoft Kinect 2 and low-cost alternatives like the Intel RealSense. Low-cost sensors often produce "Low-Quality" (LQ) images characterized by structural artifacts, non-Gaussian noise, and "black holes"—regions where depth information is missing due to surface reflectivity or sensor limitations. Enhancing these images typically requires paired datasets for supervised learning, which are difficult to acquire and align in real-world settings. This paper presents Tri-Cycle GAN, an unsupervised framework that bridges this gap without requiring paired ground-truth data.
Methodology
The authors identify a fundamental flaw in applying standard Cycle-GAN to depth enhancement: domain asymmetry. While a "High-Quality" (HQ) image can be degraded into many possible LQ versions (one-to-many), the reverse is not true. Standard cycle-consistency ($F(G(x)) = x$) forces the network to preserve noise patterns to satisfy the identity requirement, which mathematically hinders actual enhancement.
The Tri-Cycle Loss
To address asymmetry, the authors introduce the Tri-Cycle loss. Instead of a simple round-trip ($L \to H \to L$), they propose a three-step cycle: $L \to H \to L \to H$. The constraint is that the second HQ output must match the first: $G(F(G(Z_L))) = G(Z_L)$. This allows the degrader ($F$) to be one-to-many while ensuring the enhancer ($G$) remains consistent and focused on producing stable, clean depth maps.
Architectural Improvements
- U-Net Generator: A deeper and wider U-Net architecture replaces the standard generator, providing the capacity to fill large "black holes" and resolve complex scene geometry.
- Masked Similarity Loss: To handle invalid pixels (zeros) common in depth maps, the authors implement masked losses. This ensures the network only calculates loss on valid data regions, preventing it from trying to reconstruct random noise patterns found in the HQ domain.
Experimental Evaluation
The framework was validated using three distinct datasets:
- Synthetic Dataset: 120,000 images with simulated real-world noise (structural noise and adaptive holes).
- Office Dataset: A truly unsupervised set of ~1,000 unpaired images from RealSense R200 (LQ) and Kinect 2 (HQ).
- DROT Dataset: A small, publicly available dataset of 112 nearly-aligned image sets used for quantitative benchmarking.
Results
Tri-Cycle GAN demonstrates superior performance over both vanilla Cycle-GAN and non-guided methods like Sparse Depth Sensing (SDS). Using Patch-based Normalized Cross-Correlation (PNCC), the method achieved a score of 0.879 on synthetic data (vs. 0.668 for base Cycle-GAN) and 0.633 on the DROT dataset (vs. 0.213 for base Cycle-GAN).
Qualitatively, the model produces sharper object boundaries and more realistic hole-filling. By relaxing the strict cycle-consistency, the network successfully ignores the specific noise of the LQ sensor while adopting the superior geometric representation of the HQ domain.
Conclusion
Tri-Cycle GAN provides a robust, unsupervised solution for enhancing depth images from low-cost sensors. By reformulating cycle-consistency to accommodate domain asymmetry, the method achieves state-of-the-art results on real-world data. Future work will explore applying this "Tri-Cycle" logic to other asymmetric translation tasks, such as super-resolution or image dehazing, where the degradation process is inherently lossy.
Summary updated