TL;DR
This paper introduces DualSR, a novel blind super-resolution method that tackles the challenge of reconstructing high-resolution images from real-world low-resolution inputs, which often suffer from complex and unknown degradation processes. DualSR employs a dual-path architecture trained end-to-end on image patches, learning an image-specific degradation model and upsampler simultaneously using cycle consistency and a new masked interpolation loss, achieving state-of-the-art results on challenging benchmarks.
What Problem Does This Paper Address?
Single Image Super-Resolution (SISR) aims to enhance the resolution of an image. While deep learning methods have achieved remarkable success, they typically rely on training with images degraded by ideal kernels (like bicubic downsampling). This approach fails when applied to real-world images, which are subject to more complex, varied, and unknown degradation processes (e.g., varying blur kernels, noise, sensor artifacts). These real-world degradations can lead to blurry outputs and unpleasant artifacts. Existing blind SR methods often struggle with accurately estimating these complex degradations, impacting the quality of the super-resolved image. The core problem is developing an SR method that can adapt to diverse, unknown degradation kernels on a per-image basis without requiring paired training data.
Methods
DualSR proposes a dual-path architecture inspired by CycleGAN and DualGAN for image-to-image translation. It consists of two main components:
- Downsampler (GDN): This network learns to estimate the degradation process of the input LR image. It's trained using a KernelGAN-based approach to learn the patch distribution of the input image, aiming to downsample the input such that its distribution resembles the original LR image.
- Upsampler (GUP): This network learns to super-resolve the input LR image.
These two networks are trained simultaneously to improve each other. The training is driven by several loss functions:
- Adversarial Loss (LGAN): This is used to train the downsampler (GDN) and its discriminator (DDN) to distinguish between real LR image patches and those generated by GDN.
- Cycle Consistency Losses (Lcycle): These losses enforce that the upsampler and downsampler are inverses of each other. A "forward cycle" involves upsampling then downsampling, aiming to recover the original image. A "backward cycle" involves downsampling then upsampling, aiming to recover the original LR image.
- Masked Interpolation Loss (Linterp): This is a novel contribution. It addresses the issue of ringing artifacts and over-sharpening that can occur in SR, especially in low-frequency areas. It encourages the upsampler to follow bicubic interpolation only in low-frequency regions of the image, identified by a frequency mask derived from the Sobel operator.
Crucially, DualSR is trained from scratch for each input image using only patches from that image, making it highly adaptive to unknown degradation kernels.
Datasets
The paper evaluates DualSR on several datasets:
- DIV2KRK [2]: Used for quantitative evaluation, with LR images synthesized by convolving with an anisotropic Gaussian blur kernel and adding uniform multiplicative noise.
- Urban100 [15]: Similar synthesis process to DIV2KRK.
- NTIRE2017 [27]: A benchmark with a more complex and unknown downsampling operator.
- RealSR [5]: A dataset of real-world LR images captured by DSLR cameras, used for qualitative evaluation due to the lack of ground truth.
Key Insights and Ideas
- Per-Image Adaptation is Key for Real-World SR: The paper strongly emphasizes that fixed degradation models are insufficient for real-world images. DualSR's ability to learn an image-specific degradation model and upsampler from scratch on the input image is its core strength.
- Dual Learning for SR: The simultaneous training of upsampler and downsampler, where each improves the other through cycle consistency, is a powerful concept. It leverages the inherent inverse relationship between degradation and restoration.
- Targeted Interpolation for Artifact Reduction: The masked interpolation loss is a clever way to prevent blurriness in low-frequency areas while simultaneously suppressing artifacts around edges, a common problem in SR. This targeted approach is more effective than uniform interpolation.
- Blind SR Can Outperform Bicubic-Trained Methods on Real Data: Even though DualSR is a blind SR method, its ability to adapt to real-world degradations allows it to significantly outperform methods trained on ideal bicubic kernels, especially on challenging real-world datasets.
Key Results
- Quantitative Performance: DualSR achieves state-of-the-art or near state-of-the-art results on synthetic benchmarks like DIV2KRK and Urban100, outperforming recent blind and non-blind SR methods in terms of PSNR and SSIM. For instance, on DIV2KRK, it achieves a PSNR of 31.36/0.8720 and on Urban100, 25.18/0.7742. It also shows strong performance on the NTIRE2017 benchmark, outperforming BlindSR in cases where the degradation is not Gaussian.
- Qualitative Performance: On real-world images (RealSR dataset), DualSR produces visually pleasing, artifact-free, and photo-realistic results, significantly outperforming methods like bicubic interpolation, SAN+, KernelGAN+ZSSR, and BlindSR, which exhibit blurriness, ringing artifacts, or over-sharpening.
- Efficiency: The training time for DualSR is competitive, with an average training+inference time of 233 seconds on an RTX 2080 Ti GPU, which is faster than some other blind SR methods.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while DualSR performs well on real-world images, quantitative evaluation is challenging due to the lack of ground truth.
- The performance of KernelGAN, which is used for degradation estimation, can be inferior when the degradation process is very complex and not well-estimated.
- Future Work:
- The authors suggest extending DualSR to larger scale factors.
- They also propose exploring more challenging use-cases with extreme degradation settings.
- Potential applications in domains like medical imaging or electron microscopy are mentioned, where hallucination must be avoided, and high-resolution generation is critical.
Summary updated