Blind Image Super-resolution with Rich Texture-Aware Codebook

Rui Qin, Ming Sun, Fangyuan Zhang, Xing Wen, Bin Wang

TL;DR

RTCNet is a new framework for "blind" image super-resolution that fixes the common problem of "texture confusion" caused by heavy noise and blur. By training a hierarchical codebook that understands both low- and high-resolution textures simultaneously, and introducing a patch-level semantic prior, it outperforms current state-of-the-art methods by up to 0.46dB in PSNR while producing much cleaner, more realistic details.

What Problem Does This Paper Address?

Super-resolution (SR) is the art of "enhancing" a low-resolution (LR) image into a high-resolution (HR) one. "Blind" SR is the "hard mode" of this field: the computer doesn't know exactly how the image was damaged (the degradation)—it could be a mix of weird blurs, sensor noise, or heavy JPEG compression.

Recent top-tier methods use codebooks—essentially a digital library of high-quality texture "bricks" learned from HR images. When the AI sees a blurry patch, it looks up the closest match in its library to reconstruct the detail.

However, the authors identify two major flaws in current codebook methods:

  1. The Many-to-One Trap: Severe degradation can make two totally different HR textures (like grass vs. a brick wall) look nearly identical in low resolution. If the codebook was only trained on "perfect" HR data, the AI gets confused and picks the wrong "brick."
  2. The Global Semantic Bias: Most models use "priors" (pre-trained knowledge) from image classifiers like VGG. These models are great at saying "this is a dog," but they ignore local textures. A dog’s wet nose and its fluffy fur are semantically "dog," but they require very different texture reconstructions.

Methods

The authors propose RTCNet (Rich Texture-aware Codebook-based Network), which introduces two novel modules:

1. Degradation-robust Texture Prior Module (DTPM)

Instead of just learning what HR textures look like, DTPM learns the relationship between LR and HR textures during the codebook training phase.

  • Cross-Resolution Consistency: It forces the model to ensure that a degraded LR patch and its HR counterpart map to the same entry in the codebook.
  • Hierarchical Structure: It uses a "Global Scale" codebook for large patterns and a "Local Scale" codebook for minute details. This prevents the "codebook collapse" seen in simpler models where only a few entries are ever used.

2. Patch-aware Texture Prior Module (PTPM)

To fix the "global bias" problem, the authors built a new prior focused entirely on local patches.

  • Texture-Friendly Labels: They took a massive dataset and reorganized the labels using clustering (t-SNE). Instead of "Dog" or "Car," the labels are based on visual similarity (e.g., "fur-like," "mesh-like").
  • Patch-Level Pre-training: The model is trained to recognize these local textures rather than whole objects, making it much more sensitive to the fine-grained details needed for high-quality reconstruction.

Datasets

  • Training: A massive composite of DIV2K, DIV8K, Flickr2K, and OST.
  • PTPM Pre-training: They used ADE20K (semantic segmentation) and SIM (image matting) to create a specialized dataset of 17,880 texture-rich patches.
  • Evaluation: The model was tested on six standard benchmarks: DIV2K, Set14, Set5, Urban100, BSD100, and Manga109.

Key Insights and Ideas

  • LR Data Belongs in the Codebook: The most surprising insight is that training a codebook on HR data only is a mistake for blind SR. By including LR data in the codebook learning process, the model becomes "degradation-robust," learning to navigate the confusion caused by noise.
  • Semantics vs. Textures: The paper argues that high-level "ImageNet" labels are actually a distraction for low-level tasks. A "sky" patch and a "water" patch might look identical at a local level; forcing the model to distinguish them based on global context can actually lead to artifacts.
  • Deep-to-Shallow Training: To make the hierarchical codebook work, they use a sequential training strategy—training the global "coarse" details first before moving to the local "fine" details. This stabilizes a training process that is notoriously prone to failing.

Key Results

RTCNet sets a new bar for blind SR:

  • Quantitative Gains: It outperformed 10 recent state-of-the-art methods (including Real-ESRGAN and FeMaSR) across all six benchmarks, with PSNR improvements ranging from 0.16dB to 0.46dB.
  • Qualitative Superiority: Visually, RTCNet is much better at distinguishing noise from texture. While other models often "smear" fine details or create "hallucinated" artifacts, RTCNet produces sharp, believable textures (see the fur and fabric examples in Figure 7 and 16).
  • Codebook Efficiency: In testing, RTCNet utilized nearly 77% of its codebook entries, compared to just 3.2% for previous methods like FeMaSR. This proves the model is actually using its "library" of textures rather than relying on a few generic patterns.

Limitations and Future Work

  • Regular Structures: The authors admit RTCNet still struggles with highly regular, geometric textures, such as the repetitive windows on distant skyscrapers.
  • Light Degradation: The performance jump is most dramatic on "heavy" damage. For images that are only slightly blurry, the improvement over existing methods is more marginal.
  • Dataset Refinement: The authors suggest that even larger, more refined "texture-only" datasets could further improve the PTPM prior, moving further away from the limitations of object-based classification.

Summary updated

Abstract

Blind super-resolution (BSR) methods based on high-resolution (HR) reconstruction codebooks have achieved promising results in recent years. However, we find that a codebook based on HR reconstruction may not effectively capture the complex correlations between low-resolution (LR) and HR images. In detail, multiple HR images may produce similar LR versions due to complex blind degradations, causing the HR-dependent only codebooks having limited texture diversity when faced with confusing LR inputs. To alleviate this problem, we propose the Rich Texture-aware Codebook-based Network (RTCNet), which consists of the Degradation-robust Texture Prior Module (DTPM) and the Patch-aware Texture Prior Module (PTPM). DTPM effectively mines the cross-resolution correlation of textures between LR and HR images by exploiting the cross-resolution correspondence of textures. PTPM uses patch-wise semantic pre-training to correct the misperception of texture similarity in the high-level semantic regularization. By taking advantage of this, RTCNet effectively gets rid of the misalignment of confusing textures between HR and LR in the BSR scenarios. Experiments show that RTCNet outperforms state-of-the-art methods on various benchmarks by up to 0.16 ~ 0.46dB.
BibTeX
@inproceedings{qin2023blind,
  author = {Rui Qin and Ming Sun and Fangyuan Zhang and Xing Wen and Bin Wang},
  title = {{Blind Image Super-resolution with Rich Texture-Aware Codebook}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611917},
}

Loading related content...

Figure