Task Decoupled Framework for Reference-based Super-Resolution

Yixuan Huang, Xiaoyun Zhang, Yu Fu, Siheng Chen, Ya Zhang, Yanfeng Wang, Dazhi He

Introduction

Super-resolution (SR) is a fundamental task in computer vision, focused on reconstructing high-resolution (HR) images from low-resolution (LR) counterparts. While Single-Image Super-Resolution (SISR) relies on learned image priors, Reference-based Super-Resolution (RefSR) utilizes an additional HR reference image to provide authentic textures. Despite its potential, RefSR faces a significant challenge: the "coupling" of LR and reference features. When these features are processed simultaneously, models often fail to distinguish between relevant and irrelevant information, leading to suboptimal reconstruction.

The Coupling Dilemma: Underuse and Misuse

Existing RefSR methods typically concatenate LR and HR features early in the network. This coupling results in two primary failure modes:

  1. Reference-underuse: The model produces a blurry compromise between the LR input and the HR reference, failing to fully exploit the available high-frequency details even when the reference is a near-perfect match.
  2. Reference-misuse: When the reference image is semantically irrelevant to the input, the model may "hallucinate" textures from the reference onto the output, creating unnatural artifacts.
Comparison showing how the proposed method avoids reference-underuse, capturing sharper details than previous SOTA methods.

The Task Decoupled Framework

To address these issues, the authors propose a framework that decouples the SR task into three distinct stages:

Stage 1: Structural Super-Resolution

The process begins by upscaling the LR image using a standard SISR backbone (Residual Residual Dense Blocks). This stage focuses exclusively on recovering the global structure and content of the image, ensuring a solid foundation without interference from the reference image.

Stage 2: Dual-Alignment Texture Extraction

To prepare the reference textures, the model employs a hybrid alignment strategy. It uses Flow-based Alignment to correct for global shifts and Deformable Alignment (via DCN) to handle fine-grained local variations. This ensures that the extracted textures are spatially consistent with the upscaled structure from Stage 1.

Stage 3: Adaptive Texture Transfer

The final stage involves "painting" the aligned textures onto the structural base. A critical innovation here is the use of a confidence map. By calculating the similarity between the input and reference features, the model can adaptively decide how much texture to transfer. If the reference is irrelevant, the confidence map suppresses these features, effectively acting as a safety valve against artifacts.

Illustration of the reference-misuse issue. The proposed framework maintains robustness and suppresses artifacts when the reference image is irrelevant.

Experimental Results and Impact

The Task Decoupled Framework sets a new state-of-the-art (SOTA) across several benchmarks. On the CUFED5 dataset, it outperformed the previous leading method, $C^2$-Matching, by 0.4dB in PSNR. The gains were even more pronounced on datasets like Urban100 and Manga109, where improvements exceeded 1dB.

Qualitatively, the method produces significantly sharper results. As shown in the figures, it successfully restores intricate details—such as architectural textures and fine text—that competing models either blur or distort. Most importantly, the framework demonstrates remarkable robustness; it maintains high performance even when provided with low-similarity references, where traditional RefSR models often perform worse than simple SISR.

Conclusion

By decoupling structure recovery from texture transfer, this framework overcomes the long-standing issues of reference underuse and misuse in RefSR. While the multi-stage approach and dual-alignment modules introduce some computational complexity, the resulting gains in visual quality and robustness represent a significant advancement for the field. Future work may focus on optimizing the efficiency of these modules and further refining the texture extraction process.

Summary updated

Abstract

Reference-based super-resolution(RefSR) has achieved impressive progress on the recovery of high-frequency details thanks to an additional reference high-resolution(HR) image input. Although the superiority compared with Single-Image Super-Resolution(SISR), existing RefSR methods easily result in the reference-underuse issue and the reference-misuse as shown in Fig. I. In this work, we deeply investigate the cause of the two issues and further propose a novel framework to mitigate them. Our studies find that the issues are mostly due to the improper coupled framework design of current methods. Those methods conduct the super-resolution task of the input low-resolution(LR) image and the texture transfer task from the reference image together in one module, easily introducing the interference between LR and reference features. Inspired by this finding, we propose a novel framework, which decouples the two tasks of RefSR, eliminating the interference between the LR image and the reference image. The super-resolution task upsamples the LR image leveraging only the LR image itself. The texture transfer task extracts and transfers abundant textures from the reference image to the coarsely upsampled result of the super-resolution task. Extensive experiments demonstrate clear improvements in both quantitative and qualitative evaluations over state-of-the-art methods.
BibTeX
@inproceedings{huang2022task,
  author = {Yixuan Huang and Xiaoyun Zhang and Yu Fu and Siheng Chen and Ya Zhang and Yanfeng Wang and Dazhi He},
  title = {{Task Decoupled Framework for Reference-based Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00584},
}
Figure