Abstract
Recently deep learning techniques have significantly advanced image super-resolution (SR). Due to the black-box nature, quantifying reconstruction uncertainty is crucial when employing these deep SR networks. Previous approaches for SR uncertainty estimation mostly focus on capturing pixel-wise uncertainty in the spatial domain. SR uncertainty in the frequency domain which is highly related to image SR is seldom explored. In this paper, we propose to quantify spectral Bayesian uncertainty in image SR. To achieve this, a Dual-Domain Learning (DDL) framework is first proposed. Combined with Bayesian approaches, the DDL model is able to estimate spectral uncertainty accurately, enabling a reliability assessment for high frequencies reasoning from the frequency domain perspective. Extensive experiments under non-ideal premises are conducted and demonstrate the effectiveness of the proposed spectral uncertainty. Furthermore, we propose a novel Spectral Uncertainty based Decoupled Frequency (SUDF) training scheme for perceptual SR. Experimental results show the proposed SUDF can evidently boost perceptual quality of SR results without sacrificing much pixel accuracy.
Introduction
Deep learning has enabled significant progress in Super-Resolution (SR), the process of upscaling low-resolution (LR) images to high-resolution (HR) counterparts. However, these models often operate as "black boxes," making it difficult to assess the reliability of the high-frequency details they generate. In critical applications like medical imaging or surveillance, understanding whether reconstructed features are genuine or artifacts is paramount.
Traditional uncertainty estimation methods typically focus on spatial pixel-wise uncertainty. However, these are often sensitive to minor spatial misalignments. Since SR is fundamentally a task of recovering high-frequency (HF) information from low-frequency (LF) data, this research argues that reliability is best assessed in the frequency domain.
Methodology
The proposed approach introduces two primary technical innovations: Dual-Domain Learning and Spectral Bayesian Uncertainty.
Dual-Domain Learning (DDL)
The researchers modified standard architectures, such as EDSR, by incorporating parallel "frequency branches." These branches utilize complex-valued neural network layers to explicitly learn and reconstruct image spectra alongside spatial pixel data. This dual-domain approach ensures the model captures both structural and spectral characteristics of the image.
Spectral Bayesian Uncertainty and SUDF Training
By employing MC-dropout, the model generates a distribution of possible outputs. Uncertainty is quantified in the frequency domain using the Coefficient of Variation (CV), which provides a normalized metric across varying frequency ranges.
Building on this, the Spectral Uncertainty based Decoupled Frequency (SUDF) training scheme was developed. This three-step process involves:
- Training a Bayesian DDL model to generate spectral uncertainty maps.
- Training a PSNR-oriented model for high-confidence regions.
- Fine-tuning a Generative Adversarial Network (GAN) specifically for high-uncertainty, high-frequency components (e.g., stochastic textures like foliage) to enhance perceptual realism.
Results and Discussion
Experiments conducted on standard benchmarks (DIV2K, Set5, Urban100, etc.) demonstrate that the DDL-EDSR model consistently outperforms baseline models. For instance, on the Urban100 dataset (x4), PSNR improved from 25.965 to 26.342.
Key findings include:
- Texture Reliability: High-frequency components with low uncertainty typically correspond to periodic structures (e.g., architectural lines), while high uncertainty marks complex, stochastic textures.
- Robustness to Degradation: Spectral uncertainty serves as a sensitive indicator for image corruption or adversarial attacks, outperforming spatial methods in identifying unreliable outputs.
- Perception-Distortion Trade-off: The SUDF scheme successfully produces sharper, more realistic images without the significant drop in mathematical accuracy (PSNR) often seen in perceptual SR methods.
Conclusion
Quantifying uncertainty in the frequency domain provides a more global and robust assessment of SR model reliability. While the current multi-step training and Bayesian sampling increase computational complexity, the framework offers a promising path for enhancing the trustworthiness of deep learning models in inverse imaging problems such as deblurring and medical reconstruction.
Summary updated