Generalized Expectation Maximization Framework for Blind Image Super Resolution

Yuxiao Li, Yuxiao Li 0001, Zhiming Wang, Yuan ShenarXiv 2023 · DOI · arXiv

Abstract

Blind Single Image Super-Resolution (SISR) is a challenging task where the degradation kernel is unknown. Traditional methods often rely on a two-step pipeline—kernel estimation followed by restoration—which is prone to error accumulation. We present SREMN, a framework based on Generalized Expectation-Maximization (GEM) that treats the blur kernel as a latent variable. By employing a mixture of Gaussian kernels, SREMN jointly optimizes kernel estimation (E-step) and image restoration (M-step). This probabilistic approach enables semi-supervised learning from both labeled and unlabeled data, achieving state-of-the-art performance on benchmarks like DIV2KRK and Set5 while maintaining robustness across varying blur intensities.

Introduction

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a single low-resolution (LR) counterpart. While standard algorithms often assume a simple bicubic downsampling, real-world images suffer from complex, unknown "blind" blurs. The traditional approach to blind SISR involves a sequential pipeline: first estimating the blur kernel and then using it to restore the image. However, minor inaccuracies in kernel estimation can significantly degrade the final restoration. SREMN addresses this by framing the problem as a joint optimization process within a Bayesian framework.

Methodology

The core of SREMN is the Generalized Expectation Maximization (GEM) algorithm, which treats the unknown blur kernel as a latent variable. Instead of a single fixed kernel, the model utilizes a mixture of Gaussian kernels with varying bandwidths to capture complex degradations.

The SREMN framework architecture, illustrating the variational (E-step) and generative (M-step) components.
  1. E-Step (Expectation): An "E-Net" estimates the distribution of the blur kernel bandwidths from the input LR image.
  2. M-Step (Maximization): An "M-Net" takes the LR image and the estimated kernel features to reconstruct the HR image by maximizing the likelihood of the data.

This unified architecture allows for semi-supervised learning. The model is trained using a supervised loss when HR ground truths are available and an unsupervised GEM-based loss to learn from unpaired LR images, effectively leveraging real-world data that lacks high-resolution counterparts.

Experimental Setup

The researchers trained SREMN on 3,450 HR images from the DIV2K and Flick2K datasets. LR versions were synthesized using anisotropic Gaussian kernels to simulate realistic blur. Evaluation was performed on the DIV2KRK benchmark, designed for blind SR with random kernels, and the Set5 dataset for qualitative assessment across different blur bandwidths.

Key Results

SREMN achieves performance comparable to or exceeding state-of-the-art methods like DAN. It demonstrates remarkable robustness, remaining stable across a wide range of blur bandwidths (1.8 to 3.2), where other methods often falter as blur intensity increases.

Qualitative super-resolution result produced by SREMN, demonstrating sharp texture reconstruction in the eye region.

A key insight from the study is the effectiveness of semi-supervised gains. In tests with varying unsupervised rates ($\eta$), the model's performance improved as more unlabeled data was incorporated. For instance, increasing $\eta$ from 0 (fully supervised) to 0.8 (mostly unlabeled) improved the PSNR from 30.42 to 30.72, highlighting that unlabeled LR images are a valuable asset for training.

Limitations and Future Work

The authors noted that their quantitative results were somewhat constrained by hardware limitations, specifically training on a single RTX 1080Ti compared to the multi-GPU clusters used by competitors. This limited the batch size and total training compute available during development.

Future research will focus on extending the GEM-based framework to other image enhancement tasks, such as denoising and deraining. Additionally, the team aims to develop a more flexible framework capable of integrating multiple related image processing tasks into a single unified model.

Summary updated

Abstract

Learning-based methods for blind single image super resolution (SISR) conduct the restoration by a learned mapping between high-resolution (HR) images and their low-resolution (LR) counterparts degraded with arbitrary blur kernels. However, these methods mostly require an independent step to estimate the blur kernel, leading to error accumulation between steps. We propose an end-to-end learning framework for the blind SISR problem, which enables image restoration within a unified Bayesian framework with either full- or semi-supervision. The proposed method, namely SREMN, integrates learning techniques into the generalized expectation-maximization (GEM) algorithm and infers HR images from the maximum likelihood estimation (MLE). Extensive experiments show the superiority of the proposed method with comparison to existing work and novelty in semi-supervised learning.
BibTeX
@misc{li2023generalized,
  author = {Yuxiao Li and Yuxiao Li and Zhiming Wang and Yuan Shen},
  title = {{Generalized Expectation Maximization Framework for Blind Image Super Resolution}},
  year = {2023},
  eprint = {2305.13880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13880},
}
Figure