Learning the Non-Differentiable Optimization for Blind Super-Resolution

Zheng Hui, Jie Li, Xiumei Wang, Xinbo GaoCVPR 2021 · DOI

Abstract

Blind super-resolution (SR) is a challenging task due to unknown degradation processes and the limitations of standard differentiable loss functions in capturing human perception. This paper introduces AMNet-RL, a novel framework that combines a lightweight Adaptive Modulation Network (AMNet) with Deep Reinforcement Learning (RL). By utilizing a new Adaptive Modulation Layer (AMLayer) to treat blur kernels as style codes, the network efficiently adapts to various blur levels. Furthermore, the RL-based optimization allows the model to maximize non-differentiable perceptual metrics like NIQE. AMNet-RL achieves state-of-the-art visual quality with significantly faster inference speeds compared to traditional iterative methods.

Introduction

Super-resolution aims to recover high-resolution (HR) images from low-resolution (LR) counterparts. In real-world scenarios, the specific blur kernel (degradation) is often unknown, a problem known as blind SR. Current state-of-the-art solutions typically rely on iterative processes that are computationally expensive. Moreover, standard neural networks are restricted to differentiable loss functions (e.g., L1 or MSE), which often fail to align with human visual perception. Perceptual metrics like the Natural Image Quality Evaluator (NIQE) are non-differentiable, preventing direct optimization via standard backpropagation. This paper addresses these challenges by proposing an efficient architecture and an RL-based training strategy.

Methodology

Adaptive Modulation Network (AMNet)

To improve efficiency over existing Spatial Feature Transform (SFT) methods, the authors introduce the AMLayer. Inspired by Adaptive Instance Normalization (AdaIN) used in style transfer, the AMLayer treats the blur kernel as a global "style" code. It modulates the mean and standard deviation of image features to account for different blur levels. This approach allows the network to remain lightweight while effectively handling diverse degradations.

Overview of the AMNet architecture, the Adaptive Modulation Layer (AMLayer), and the Adaptive Modulation Residual Block (AMRB).

Reinforcement Learning Optimization (AMNet-RL)

To optimize for non-differentiable metrics, the authors frame kernel estimation as a Reinforcement Learning problem using the Deep Deterministic Policy Gradient (DDPG) algorithm. The framework consists of:

  • Actor: A kernel predictor that estimates the blur kernel from the LR image.
  • Renderer: A pre-trained AMNet that generates the SR image based on the predicted kernel.
  • Reward: The improvement in a target metric (e.g., NIQE or PSNR) relative to a baseline.
  • Critic: A network that learns to estimate expected rewards, providing a gradient to train the Actor.

This setup enables the Actor to learn kernel predictions that maximize perceptual quality, effectively bridging the gap for non-differentiable quality assessment.

The AMNet-RL framework, illustrating the interaction between the Actor, Critic, and Environment for optimizing non-differentiable metrics.

Experiments and Results

Datasets and Training

The model was trained on 3,450 high-quality images from DIV2K and Flickr2K. Evaluation was performed on standard benchmarks including Set5, Set14, BSD100, Urban100, Manga109, and PIRM Val, using LR images synthesized with isotropic Gaussian blur kernels (widths 0.2 to 4.0).

Performance and Speed

AMNet-RL demonstrates a significant speed advantage, running approximately 11x faster than Iterative Kernel Correction (IKC) and 4x faster than the Deep Alternating Network (DAN). Unlike previous multi-step RL restoration methods, this framework uses a single-step action, making it suitable for real-time applications.

Quality and Fidelity

In terms of fidelity, the AMNet-L variant outperformed non-blind methods like SRMDNF and SFTMD in PSNR on the Set5 dataset. When optimized for perception (AMGAN-RL), the model achieved the best (lowest) NIQE scores across multiple datasets, producing sharper edges and more realistic textures compared to USRGAN.

Conclusion

AMNet-RL provides a powerful and efficient solution for blind super-resolution. By treating blur as a style and leveraging reinforcement learning for non-differentiable optimization, the framework overcomes the limitations of traditional iterative and gradient-based methods. Future research could extend this approach to more complex, anisotropic blur kernels and other non-differentiable image processing tasks.

Summary updated

BibTeX
@inproceedings{hui2021learning,
  author = {Zheng Hui and Jie Li and Xiumei Wang and Xinbo Gao},
  title = {{Learning the Non-Differentiable Optimization for Blind Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00213},
}
Figure