Blind Image Super-resolution with Rich Texture-Aware Codebooks

Rui Qin, Ming Sun, Fangyuan Zhang, Xing Wen, Bin Wang

Abstract: Blind super-resolution (BSR) methods based on high-resolution (HR) reconstruction codebooks have achieved promising results in recent years. However, we find that a codebook based on HR reconstruction may not effectively capture the complex correlations between low-resolution (LR) and HR images. In detail, multiple HR images may produce similar LR versions due to complex blind degradations, causing the HR-dependent only codebooks having limited texture diversity when faced with confusing LR inputs. To alleviate this problem, we propose the Rich Texture-aware Codebook-based Network (RTCNet), which consists of the Degradation-robust Texture Prior Module (DTPM) and the Patch-aware Texture Prior Module (PTPM). DTPM effectively mines the cross-resolution correlation of textures between LR and HR images by exploiting the cross-resolution correspondence of textures. PTPM uses patch-wise semantic pre-training to correct the misperception of texture similarity in the high-level semantic regularization. By taking advantage of this, RTCNet effectively gets rid of the misalignment of confusing textures between HR and LR in the BSR scenarios. Experiments show that RTCNet outperforms state-of-the-art methods on various benchmarks by up to 0.16 ~ 0.46dB.

The Challenges of Blind Super-Resolution

Super-resolution (SR) aims to enhance a low-resolution (LR) image into a high-resolution (HR) counterpart. "Blind" SR is particularly difficult because the degradation process—a combination of blur, sensor noise, and compression—is unknown. Recent state-of-the-art methods utilize codebooks, which are digital libraries of high-quality texture "bricks" learned from HR images. When the AI encounters a blurry patch, it attempts to find the closest match in its library to reconstruct the detail.

However, current codebook methods suffer from two major flaws:

  1. The Many-to-One Trap: Severe degradation can make two distinct HR textures (e.g., grass vs. a brick wall) look nearly identical in low resolution. If a codebook is trained only on perfect HR data, the model often selects the wrong texture entry when faced with ambiguous LR inputs.
  2. Global Semantic Bias: Most models rely on priors from image classifiers like VGG. While these models excel at object recognition (e.g., "this is a dog"), they ignore local texture nuances. A dog’s nose and its fur are semantically identical to a classifier but require vastly different texture reconstructions.
The Many-to-One Trap: Similar LR patches (LR 1 and LR 2) can originate from very different HR textures (HR 1 and HR 2), leading to reconstruction errors in standard codebooks.

The RTCNet Framework

To address these issues, the authors propose RTCNet (Rich Texture-aware Codebook-based Network), featuring two novel modules designed to improve texture fidelity.

Degradation-robust Texture Prior Module (DTPM)

Instead of learning only HR textures, DTPM learns the relationship between LR and HR textures during the codebook training phase. It enforces Cross-Resolution Consistency, ensuring that a degraded LR patch and its HR counterpart map to the same codebook entry. Furthermore, it employs a hierarchical structure—using a "Global Scale" codebook for large patterns and a "Local Scale" codebook for fine details—to prevent "codebook collapse," where only a few entries are utilized by the model.

Patch-aware Texture Prior Module (PTPM)

To overcome global semantic bias, PTPM introduces a prior focused on local patches rather than whole objects. The authors reorganized a massive dataset using clustering to create Texture-Friendly Labels based on visual similarity (e.g., "fur-like" or "mesh-like"). By pre-training the model to recognize these local textures, RTCNet becomes significantly more sensitive to the fine-grained details necessary for high-quality reconstruction.

Comparison of Priors: Traditional classification-based priors (top) focus on global shapes, whereas the proposed Patch-aware Texture Prior (bottom) focuses on local texture similarity, distinguishing between face, cloth, and fur.

Experimental Results and Insights

RTCNet was evaluated on standard benchmarks including DIV2K, Set14, and Urban100. The model demonstrated significant improvements over 10 recent state-of-the-art methods, such as Real-ESRGAN and FeMaSR.

  • Quantitative Gains: RTCNet achieved PSNR improvements ranging from 0.16dB to 0.46dB across all benchmarks.
  • Qualitative Superiority: Visually, the model is superior at distinguishing noise from actual texture, producing sharp, believable details in fur and fabric where other models often create artifacts or "smear" the image.
  • Codebook Efficiency: Testing revealed that RTCNet utilized nearly 77% of its codebook entries, a massive increase over the 3.2% utilization seen in previous methods like FeMaSR. This indicates that the model is effectively leveraging its entire library of learned textures.

Conclusion

RTCNet demonstrates that integrating LR data into the codebook learning process and shifting from global to patch-level semantic priors are critical for solving blind SR. While the model still faces challenges with highly regular geometric structures (like distant skyscraper windows), it sets a new standard for reconstructing clean, realistic textures from heavily degraded images.

Summary updated

Abstract

Blind super-resolution (BSR) methods based on high-resolution (HR) reconstruction codebooks have achieved promising results in recent years. However, we find that a codebook based on HR reconstruction may not effectively capture the complex correlations between low-resolution (LR) and HR images. In detail, multiple HR images may produce similar LR versions due to complex blind degradations, causing the HR-dependent only codebooks having limited texture diversity when faced with confusing LR inputs. To alleviate this problem, we propose the Rich Texture-aware Codebook-based Network (RTCNet), which consists of the Degradation-robust Texture Prior Module (DTPM) and the Patch-aware Texture Prior Module (PTPM). DTPM effectively mines the cross-resolution correlation of textures between LR and HR images by exploiting the cross-resolution correspondence of textures. PTPM uses patch-wise semantic pre-training to correct the misperception of texture similarity in the high-level semantic regularization. By taking advantage of this, RTCNet effectively gets rid of the misalignment of confusing textures between HR and LR in the BSR scenarios. Experiments show that RTCNet outperforms state-of-the-art methods on various benchmarks by up to 0.16 ~ 0.46dB.
BibTeX
@inproceedings{qin2023blind,
  author = {Rui Qin and Ming Sun and Fangyuan Zhang and Xing Wen and Bin Wang},
  title = {{Blind Image Super-resolution with Rich Texture-Aware Codebooks}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611917},
}
Figure