TL;DR
This paper introduces MoESR, a novel blind super-resolution (SR) method that tackles the challenge of unknown image degradation kernels. By employing a "Mixture of Experts" approach where specialized networks handle specific kernel types, MoESR achieves state-of-the-art performance, significantly outperforming existing methods, especially for challenging ×4 SR tasks.
What Problem Does This Paper Address?
Modern deep learning-based super-resolution (SR) methods have achieved impressive results, but they typically rely on the assumption that the low-resolution (LR) input image is degraded by a known, fixed kernel (e.g., bicubic downsampling). This assumption breaks down in real-world scenarios where image degradation can vary significantly due to factors like camera imperfections, motion blur, and atmospheric conditions. Blind SR methods aim to address this by estimating the unknown degradation kernel and then performing SR. However, existing blind SR methods often compromise on quality compared to kernel-specific approaches or suffer from slow inference times and limited accuracy when the estimated kernel deviates from the ground truth. This paper aims to bridge this gap by developing a blind SR method that leverages the strengths of both specialized kernel-specific SR networks and internal image learning for improved accuracy and robustness.
Methods
The core of MoESR is a Mixture of Experts (MoE) architecture. Instead of a single network trying to handle all possible degradations, MoESR utilizes multiple "expert" networks, each trained for a specific range of blur kernel parameters.
- Kernel-Specific Experts: A set of expert networks are trained to perform SR for different, predefined blur kernels. These kernels are characterized by anisotropic Gaussian and motion blur properties, defined by eigenvalues ($\lambda_1, \lambda_2$) and a rotation angle ($\theta$).
- Kernel Estimation: To determine which expert to use for a given input image, MoESR employs a two-stage kernel estimation process:
- Image Sharpness Evaluator (ISE): This module is trained to detect blurry or over-sharpened regions in upsampled images. It learns to predict the error between upsampled images from different experts and the ground truth.
- Kernel Estimation Network (KEN): The ISE outputs for all experts are fed into KEN. KEN then estimates the most likely kernel parameters for the input image. This is a novel approach that leverages the sharpness evaluation of expert outputs to infer the kernel.
- Expert Selection and Fine-tuning: Based on the estimated kernel parameters, the most suitable kernel-specific expert is selected. Crucially, this selected expert is then fine-tuned on the input image itself. This fine-tuning step allows the model to leverage internal image statistics and adapt to subtle variations, further improving the SR quality. The fine-tuning process uses a DualSR-like pipeline for efficient adaptation.
Datasets
- Training: The paper uses 800 images from the DIV2K dataset for training the expert networks, ISE, and KEN.
- Evaluation:
- Synthetic Datasets: DIV2KRK, Flickr2KRK, and Urban100RK are used. These datasets are derived from DIV2K, Flickr2K, and Urban100, respectively, and are degraded with randomly sampled anisotropic Gaussian kernels with added multiplicative noise.
- Real-World Images: The paper also evaluates on real-world images, referencing datasets like RealSR and Set5.
Key Insights and Ideas
- Mixture of Experts for Blind SR: The core idea of using specialized experts for different kernel types, combined with a robust kernel estimation mechanism, is a significant advancement for blind SR. It allows for higher quality than a single, general-purpose blind SR network.
- Sharpness-Based Kernel Estimation: Using an Image Sharpness Evaluator (ISE) to guide kernel estimation is an innovative approach. It directly links the SR output quality to the kernel estimation process, making it more intuitive and effective.
- Internal Learning via Fine-tuning: The fine-tuning step on the test image is critical. It allows MoESR to adapt to the specific degradation of the input image, correcting for potential inaccuracies in kernel estimation and exploiting image-specific features, which is a key advantage over methods that only rely on external training.
- Kernel Estimation Accuracy: The paper demonstrates that MoESR's kernel estimation is significantly more accurate than other blind SR methods, especially for challenging ×4 SR, which directly translates to better SR performance.
Key Results
- Quantitative Performance: MoESR achieves state-of-the-art results on synthetic datasets. For the challenging ×4 SR task on the DIV2KRK dataset, MoESR shows a PSNR improvement of 0.93 dB over the next best method (DAN).
- Qualitative Performance: Visual comparisons show that MoESR generates sharper and cleaner images with fewer artifacts compared to other blind SR methods like DAN, IKC, and KernelGAN. It effectively handles subtle compression artifacts in real-world images.
- Kernel Estimation Accuracy: MoESR's estimated kernels exhibit significantly lower Sum Absolute Error (SAE) compared to other methods, particularly for ×4 SR.
- Fine-tuning Benefits: Fine-tuning with 1000 iterations provides a PSNR improvement of 0.68 dB (×2 SR) and 0.34 dB (×4 SR) on DIV2KRK, demonstrating its effectiveness.
Limitations and Future Work
- Runtime Efficiency: The paper acknowledges that MoESR's processing time (around 34-44 seconds for ×2 and ×4 SR on a single image) is slower than some degradation-aware networks like IKC and DAN, although faster than some self-supervised methods. They suggest that reducing the number of experts or fine-tuning iterations can improve speed.
- Noise Sensitivity: The authors note that MoESR performs sub-optimally when the input image contains substantial noise, suggesting that a denoising step might be necessary in such scenarios.
- Future Work: The authors suggest exploring more advanced SR networks to further improve performance and reducing the computational time per image.
Summary updated