Kernel Modeling Super-Resolution on Real Low-Resolution Images

Ruofan Zhou, Sabine SüsstrunkICCV 2019 · OpenReview

Abstract

Deep convolutional neural networks (CNNs), trained on corresponding pairs of high- and low-resolution images, achieve state-of-the-art performance in single-image super-resolution and surpass previous signal-processing based approaches. However, their performance is limited when applied to real photographs. The reason lies in their training data: low-resolution (LR) images are obtained by bicubic interpolation of the corresponding high-resolution (HR) images. The applied convolution kernel significantly differs from real-world camera-blur. Consequently, while current CNNs well super-resolve bicubic-downsampled LR images, they often fail on camera-captured LR images. To improve generalization and robustness of deep super-resolution CNNs on real photographs, we present a kernel modeling super-resolution network (KMSR) that incorporates blur-kernel modeling in the training. Our proposed KMSR consists of two stages: we first build a pool of realistic blur-kernels with a generative adversarial network (GAN) and then we train a super-resolution network with HR and corresponding LR images constructed with the generated kernels. Our extensive experimental validations demonstrate the effectiveness of our single-image super-resolution approach on photographs with unknown blur-kernels.

Introduction

Most modern CNN-based Super-Resolution (SR) methods require massive datasets of paired Low-Resolution (LR) and High-Resolution (HR) images. Because capturing perfectly aligned LR-HR image pairs in the real world is incredibly difficult, researchers typically synthesize LR images by applying a fixed mathematical operation—usually bicubic interpolation—to HR images.

The problem is that the bicubic convolution kernel looks nothing like real-world camera blur, which is a complex combination of optical blur, atmospheric conditions, camera shake, and lens aberrations. Because existing networks are trained to reverse a specific, artificial degradation, they suffer a severe drop in performance and generalize poorly when applied to actual photographs taken by real cameras.

Methodology

To bridge the gap between synthetic training data and real-world photographs, we propose a two-stage framework called KMSR:

  1. Kernel Pool Building (The GAN Stage): Instead of using a fixed bicubic kernel, we first extract real blur kernels from actual photographs using a dark-channel prior kernel estimation algorithm. Because this extraction process is computationally expensive and yields a limited variety of kernels, we train a Wasserstein GAN with Gradient Penalty (WGAN-GP) on these extracted kernels. The trained GAN acts as a generator for an infinite, highly diverse pool of realistic camera blur kernels.
  2. Super-Resolution (The CNN Stage): We utilize a standard 16-residual-block CNN for the actual super-resolution task. However, instead of training it on bicubic-downsampled images, we construct a custom training set. During training, HR patches are convolved with random realistic blur kernels sampled from the GAN-generated pool to create the corresponding LR patches.
Overview of the KMSR framework consisting of kernel pool building and super-resolution stages.

Key Insights: Modeling the Degradation

A critical insight of this work is that the standard bicubic kernel is "band-pass" (designed to artificially preserve sharpness and avoid aliasing), whereas real camera blur is "low-pass" (naturally attenuating high frequencies). Training on bicubic data teaches the network the wrong physics.

Comparison of blur kernel coefficients, highlighting the difference between the bicubic proxy and realistic kernels modeled by KMSR.

KMSR flips the standard SR paradigm: while many models use GANs to hallucinate high-frequency details in the final image (often leading to artifacts), KMSR uses a GAN purely to model the degradation (the blur kernels). This allows the SR network to be trained with a standard, stable L1 loss while still benefiting from the GAN's ability to model complex distributions. Furthermore, generative augmentation proved superior to simple data augmentation; using a GAN to generate novel kernels yielded a 0.5 dB PSNR improvement over simply rotating or flipping initially estimated kernels.

Experimental Results

We evaluated KMSR against state-of-the-art models using the DSLR Photo Enhancement Dataset (DPED) for kernels, DIV2K for source images, and a custom zoom-in dataset captured with a 24-70mm lens to provide real-world ground truth.

  • Synthetic/Gaussian Kernels: KMSR outperforms models like EDSR and DBPN by up to 1.91 dB PSNR on images degraded by standard Gaussian kernels.
  • Unseen Realistic Kernels: When tested on DIV2K images degraded by real kernels extracted from the iPhone 3GS dataset, KMSR achieved 27.52 dB PSNR (for x2 upscaling), significantly surpassing DBPN (25.30 dB).
  • Real Camera Pairs: On the custom 35mm vs. 70mm lens dataset, KMSR achieved 29.13 dB PSNR, outperforming EDSR (27.45 dB).
  • Human Evaluation: In a crowdsourced psychovisual experiment using real photographs, human evaluators preferred KMSR's output over top competitors in 44 out of 50 images (88% of the time).

Conclusion

While highly effective, KMSR currently assumes a noise-free degradation model. Real-world LR images often suffer from sensor noise in addition to blur, meaning performance may degrade on heavily noisy images. Future work will explore joint modeling of realistic noise profiles alongside blur kernels to create an even more robust real-world SR pipeline.

Summary updated

Abstract

Deep convolutional neural networks (CNNs), trained on corresponding pairs of high- and low-resolution images, achieve state-of-the-art performance in single-image super-resolution and surpass previous signal-processing based approaches. However, their performance is limited when applied to real photographs. The reason lies in their training data: low-resolution (LR) images are obtained by bicubic interpolation of the corresponding high-resolution (HR) images. The applied convolution kernel significantly differs from real-world camera-blur. Consequently, while current CNNs well super-resolve bicubic-downsampled LR images, they often fail on camera-captured LR images. To improve generalization and robustness of deep super-resolution CNNs on real photographs, we present a kernel modeling super-resolution network (KMSR) that incorporates blur-kernel modeling in the training. Our proposed KMSR consists of two stages: we first build a pool of realistic blur-kernels with a generative adversarial network (GAN) and then we train a super-resolution network with HR and corresponding LR images constructed with the generated kernels. Our extensive experimental validations demonstrate the effectiveness of our single-image super-resolution approach on photographs with unknown blur-kernels.
BibTeX
@inproceedings{zhou2019kernel,
  author = {Ruofan Zhou and Sabine Süsstrunk},
  title = {{Kernel Modeling Super-Resolution on Real Low-Resolution Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=5cNGmd6x5dj},
}
Figure