Abstract
Depth Super-Resolution (DSR) is essential for enhancing low-resolution depth maps captured by consumer-grade sensors. However, real-world degradations are often unknown and complex, causing traditional methods—which assume simple bicubic downsampling—to fail. We present DORNet, a Degradation Oriented and Regularized Network for blind DSR. DORNet employs a self-supervised degradation learning strategy to implicitly model unknown degradations and a targeted RGB-D fusion mechanism to sharpen depth maps. Experiments demonstrate that DORNet significantly outperforms state-of-the-art methods on real-world datasets while maintaining high efficiency for practical applications.
Introduction
Depth maps captured by consumer-grade sensors, such as Time-of-Flight (ToF) cameras in smartphones, are frequently low-resolution and plagued by noise. While guided DSR aims to address this by using a high-resolution RGB image as a reference, most existing methods rely on the assumption that degradation is known and follows a simple bicubic downsampling process.
In real-world scenarios, this assumption is often invalid. Physical sensors introduce complex degradations, including structural distortion, blur, and noise influenced by lighting or surface reflectivity. When traditional models encounter these "blind" (unknown) degradations, their performance degrades significantly, resulting in blurry edges and misaligned structures. DORNet addresses this challenge by learning to model unknown degradations implicitly, without requiring explicit degradation labels.
Methodology
The DORNet framework consists of two primary branches designed to handle unknown degradations and optimize feature fusion.
Self-Supervised Degradation Learning
Instead of assuming a fixed degradation model, DORNet uses a Degradation Encoder to extract an implicit representation ($D$) directly from the low-resolution input. To regularize this learning, a Router dynamically selects the most appropriate degradation kernel generators (of varying sizes) to "re-degrade" the predicted high-resolution depth. By forcing this re-degraded version to match the original low-resolution input, the model learns underlying degradation patterns in a self-supervised loop. A contrastive loss further refines this by pushing the re-degraded depth closer to the low-resolution input and away from the clean ground truth.
Degradation-Oriented Feature Transformation (DOFT)
Standard fusion methods often combine RGB and depth data indiscriminately. In contrast, the Degradation-Oriented Feature Transformation (DOFT) module uses learned degradation priors to selectively propagate information from the RGB image. It utilizes Deformable Convolutions (DCN) guided by the degradation representation to focus on RGB features that specifically sharpen edges or fill gaps where the depth signal is most degraded.
Experimental Results
DORNet was evaluated on three major benchmarks: the real-world RGB-D-D dataset, the TOFDSR dataset, and the synthetic NYU-v2 dataset.
Performance and Efficiency
On the real-world RGB-D-D dataset, DORNet achieved an RMSE of 3.42 cm, outperforming the previous state-of-the-art method, SFG (3.88 cm), and significantly surpassing SGNet (5.32 cm). Beyond accuracy, the model is highly efficient. A lightweight version, DORNet-T, achieves 161 FPS on an NVIDIA 4090 GPU, making it highly suitable for real-time AR/VR applications.
Robustness to Real-World Noise
As shown in the visual comparisons and histograms, real-world degradation differs significantly from synthetic bicubic downsampling. DORNet's ability to model these complex patterns allows it to maintain a low error rate even under varying levels of Gaussian noise, where traditional models typically fail.
Conclusion
DORNet provides a robust and efficient solution for blind Depth Super-Resolution by moving away from fixed degradation assumptions. By learning implicit degradation representations and utilizing targeted RGB-D fusion, it bridges the gap between synthetic training and real-world application. Future work may extend this degradation-oriented strategy to other multi-modal tasks, such as satellite pan-sharpening or LiDAR depth completion.
Summary updated