Denoising Diffusion Probabilistic Models for Robust Image Super-Resolution in the Wild

Hshmat Sahak, Daniel Watson, Chitwan Saharia, David J. FleetarXiv 2023 · DOI · arXiv

Abstract

Super-resolution (SR) in real-world scenarios, often termed "blind SR," remains a significant challenge due to complex, unknown degradations such as sensor noise and compression artifacts. While Diffusion Probabilistic Models have shown success in controlled SR tasks, they have historically lagged behind Generative Adversarial Networks (GANs) in handling out-of-distribution real-world data. We present SR3+, a diffusion-based approach that achieves state-of-the-art performance in blind SR. By incorporating a fully convolutional architecture, a higher-order degradation pipeline, and a novel Noise Conditioning Augmentation (NCA) technique, SR3+ produces realistic, high-fidelity textures that outperform existing GAN-based methods like Real-ESRGAN.

Introduction

Most super-resolution research assumes a "clean" environment where low-resolution (LR) images are simple bicubic downsamplings of high-resolution (HR) counterparts. However, "in the wild," images suffer from a combination of motion blur, sensor noise, and aggressive JPEG compression. This "blind" SR task requires models to be robust against unpredictable corruptions. While models like the original SR3 demonstrated the power of diffusion for clean SR, they often failed to generalize to messy real-world data. SR3+ addresses these limitations, bridging the gap between diffusion models and robust real-world performance.

Methods

SR3+ introduces three core innovations to enhance robustness and scalability:

  1. Fully Convolutional Architecture: Replacing the self-attention layers of the original SR3 with a large-scale convolutional UNet allows the model to process images of arbitrary resolutions and aspect ratios without the constraints of fixed-size attention maps.
  2. Higher-Order Degradation Pipeline: To simulate the complex history of real-world digital images, SR3+ employs a multi-stage degradation process during training. This involves applying sequences of blur, resizing, and JPEG compression multiple times.
  3. Noise Conditioning Augmentation (NCA): This technique involves adding a controlled amount of Gaussian noise to the LR input during both training and inference. By conditioning the model on this noise level, SR3+ is encouraged to rely on its learned natural image priors rather than attempting to reconstruct corrupted or noisy pixels directly.
The SR3+ training pipeline, featuring a multi-pass degradation process followed by Noise Conditioning Augmentation.

Key Results

SR3+ was evaluated on benchmarks like RealSR and DRealSR. Quantitatively, it achieved a DRealSR FID score of 32.37, a significant improvement over Real-ESRGAN's 37.22.

Qualitatively, SR3+ excels at generating realistic, stochastic textures. Unlike GANs, which can produce "plastic" or overly sharpened artifacts, SR3+ captures the natural grain and fine details of the world. The model also shows strong scaling properties; increasing the parameter count from 40M to 400M and training on a massive 61-million-image dataset resulted in consistent quality gains.

Qualitative comparison of SR3+ against Bicubic interpolation and Real-ESRGAN. SR3+ variants show superior detail recovery and more natural textures as model capacity increases.

Discussion and Limitations

Despite its strengths, SR3+ faces several challenges. Like many generative models, it struggles with text, often "hallucinating" blurry characters into meaningless but plausible-looking shapes. Additionally, the iterative nature of diffusion sampling (requiring 256 steps) makes it significantly slower than one-shot GAN models. Finally, at very high noise conditioning levels, the model may lose precise alignment with the input image. Future work will focus on improving sampling speed and text fidelity.

Conclusion

SR3+ demonstrates that diffusion models, when combined with robust degradation modeling and noise conditioning, can surpass GANs in real-world super-resolution tasks. Its ability to generate high-fidelity, natural textures marks a significant step forward for "in the wild" image enhancement.

Summary updated

BibTeX
@misc{sahak2023denoising,
  author = {Hshmat Sahak and Daniel Watson and Chitwan Saharia and David J. Fleet},
  title = {{Denoising Diffusion Probabilistic Models for Robust Image Super-Resolution in the Wild}},
  year = {2023},
  eprint = {2302.07864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.07864},
}
Figure