MMSR: Multiple-Model Learned Image Super-Resolution Benefiting from Class-Specific Image Priors

Cansu Korkmaz, A. Murat Tekalp, Zafer Dogan

TL;DR

The authors propose MMSR (Multiple-Model Super-Resolution), a framework that moves away from "one-size-fits-all" image upscaling. By training a bank of specialized models for different image types (like text or textures) and using a "fusion network" to combine their outputs, they achieve significantly sharper results and higher PSNR than generic state-of-the-art models.

What Problem Does This Paper Address?

Single-image super-resolution (SISR) is the task of reconstructing a high-resolution (HR) image from a low-resolution (LR) input. Most modern deep-learning approaches train a single, generic model on a massive, diverse dataset (like DIV2K).

The problem? Images are not monolithic. A patch of sharp, high-contrast text requires very different "priors" (learned rules about how pixels should look) than a patch of soft, organic grass or a brick wall. A generic model often compromises, leading to "ringing" artifacts around text or blurred details in complex textures. Furthermore, generic models show high performance variance; they might work brilliantly on one image but fail on another simply because the test image's characteristics don't match the training set.

Methods

The researchers introduce the Multiple-Model SR (MMSR) framework, which consists of two main stages:

  1. A Bank of Specialized SR Models: Instead of one model, they use $N$ models. $N-1$ models are trained on specific classes (e.g., one specifically for text, one for textures), and the $N^{th}$ model is a generic "catch-all" model for everything else. They used the popular EDSR (Enhanced Deep Residual Networks) architecture as the backbone for these models.
  2. A Post-SR Fusion Network: This is the "secret sauce." This network takes the outputs from all the specialized models as its input and learns how to intelligently fuse them into a single final image.

The authors tested two fusion scenarios:

  • Class-Specific Fusion: Used when the target image is homogeneous (e.g., a page of text).
  • Generic Fusion: Used for complex images containing mixed content (e.g., a photo of a sign in front of a park).

Datasets

The study utilized several key datasets to train and evaluate the specialized models:

  • ImagePairs: A dataset containing realistic text images captured via a beam-splitter camera rig.
  • DIV2K: A standard benchmark for SR containing high-quality natural images (used here for texture and generic priors).
  • RealSR-v3: A dataset of real-world scenes with sharp edges and fine textures, used to test how well the model generalizes to non-homogeneous, "messy" real-world data.
  • Standard Benchmarks: Set5, Set14, B100, and Urban100 were used to establish the baseline performance of generic models.

Key Insights and Ideas

  • The "Aha" Moment in Data: The authors used t-SNE (a visualization technique) to plot image patches. They found that text patches and texture patches formed two very distinct clusters. This mathematically confirmed that a single model trying to learn both is essentially trying to solve two different problems at once.
  • Fusion is Better than Selection: You might think the best approach is just to pick the "text model" for text. However, the authors found that the fusion network actually outperformed the best single class-specific model. Even for a pure text image, the fusion network learned to extract the best features from multiple models to produce a superior result.
  • Reducing Variance: Generic models often have a "hit or miss" quality. MMSR significantly reduces the variance in performance across different types of images, making the upscaling more reliable.

Key Results

  • Quantitative Gains: On text-heavy images, the MMSR approach achieved a PSNR of 33.668 dB, outperforming the generic EDSR (32.903 dB) and the specialized text-only model (33.328 dB).
  • Real-World Performance: On the non-homogeneous RealSR-v3 dataset, the MMSR model (using generic fusion) reached 30.279 dB, a massive jump over the generic model's 23.482 dB.
  • Qualitative Quality: Visually, the MMSR results show much sharper letter boundaries in text and more realistic grain in textures, whereas competing methods like ZSSR (Zero-Shot SR) often introduced significant "ringing" or blurring.

Limitations and Future Work

  • Computational Overhead: Running $N$ different SR models plus a fusion network is computationally more expensive than running a single model. This might limit its use in real-time mobile applications without further optimization.
  • Defining the Classes: The paper focuses on text and texture. Determining the optimal number of classes ($N$) and which specific categories (e.g., faces, foliage, architecture) provide the most benefit remains an open question.
  • Automation: Future work could involve automating the discovery of these classes or using a more dynamic "routing" mechanism to decide which models to activate for specific regions of an image.

Summary updated

Abstract

Assuming a known degradation model, the performance of a learned image super-resolution (SR) model depends on how well the variety of image characteristics within the training set matches those in the test set. As a result, the performance of an SR model varies noticeably from image to image over a test set depending on whether characteristics of specific images are similar to those in the training set or not. Hence, in general, a single SR model cannot generalize well enough for all types of image content. In this work, we show that training multiple SR models for different classes of images (e.g., for text, texture, etc.) to exploit class-specific image priors and employing a post-processing network that learns how to best fuse the outputs produced by these multiple SR models surpasses the performance of state-of-the-art generic SR models. Experimental results clearly demonstrate that the proposed multiple-model SR (MMSR) approach significantly outperforms a single pre-trained state-of-the-art SR model both quantitatively and visually. It even exceeds the performance of the best single class-specific SR model trained on similar text or texture images.
BibTeX
@inproceedings{korkmaz2022mmsr,
  author = {Cansu Korkmaz and A. Murat Tekalp and Zafer Dogan},
  title = {{MMSR: Multiple-Model Learned Image Super-Resolution Benefiting from Class-Specific Image Priors}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897278},
}

Loading related content...

Figure