Reference-based Image and Video Super-Resolution via C2-Matching

Yuming Jiang, Kelvin C. K. Chan, Xintao Wang, Chen Change Loy, Ziwei Liu

Abstract

Reference-based super-resolution (Ref-SR) leverages high-resolution (HR) textures from a reference image to enhance a low-resolution (LR) input. This task is complicated by the "transformation gap" (differences in scale and rotation) and the "resolution gap" (matching blurry LR features to sharp HR ones). We introduce $C^2$-Matching (Robust Matching Cross transformation and Cross resolution), a framework that employs contrastive learning and teacher-student correlation distillation to overcome these hurdles. $C^2$-Matching achieves state-of-the-art performance across multiple image and video benchmarks, including a new "in-the-wild" web-referenced dataset.

Introduction

Super-resolution (SR) traditionally relies on internal information from a single image or video sequence. Ref-SR offers a more powerful alternative by providing an external HR reference image. However, the effectiveness of Ref-SR depends entirely on accurate correspondence matching: finding exactly which pixel in the reference image matches which pixel in the input. Existing methods often fail when the reference image is transformed (rotated or scaled) or when the resolution difference between the input and reference is too large. $C^2$-Matching addresses these fundamental gaps to provide robust, high-quality texture restoration.

Methodology: Bridging the Gaps

The $C^2$-Matching framework introduces three core components to ensure robust feature correspondence and texture transfer:

  • Contrastive Correspondence Network: To handle the transformation gap, the network is trained using contrastive learning on augmented views. This ensures that feature descriptors remain consistent regardless of scale or rotation changes in the reference image.
  • Teacher-Student Correlation Distillation: To bridge the resolution gap, a "teacher" network matches two HR images, providing a high-confidence "ground truth" for correspondences. This knowledge is distilled into a "student" network that learns to match LR inputs to HR references, improving accuracy in complex textured regions.
  • Dynamic Aggregation Module: Moving beyond rigid patch-based matching, the system utilizes Deformable Convolutions (DCN) to flexibly gather and align textures from the reference image based on learnable offsets.
Figure 1: The C2-Matching framework establishes correspondences between a blurry low-resolution input and a sharp high-resolution reference, even under significant transformations.

Datasets and Experimental Setup

The model was evaluated on standard benchmarks including CUFED5, Sun80, Urban100, and Manga109 for images, and REDS and Vid4 for video. To test real-world robustness, the authors introduced the Webly-Referenced SR (WR-SR) dataset. Unlike previous datasets with curated references, WR-SR uses images retrieved via Google Image search, mimicking a realistic scenario where a user searches the web for a high-quality reference to fix a low-quality photo.

Key Results

$C^2$-Matching demonstrates significant quantitative and qualitative improvements:

  • Image SR: It outperforms previous state-of-the-art methods like TTSR and MASA-SR by up to 0.7dB in PSNR on the CUFED5 benchmark.
  • Video SR: On the REDS dataset, the method provides a 0.3dB boost over standard video super-resolution techniques.
  • Visual Quality: In user studies, over 80% of participants preferred $C^2$-Matching results, noting sharper edges and more realistic texture reconstruction compared to competing methods.
Figure 5: Visual comparison between the low-resolution input (top) and the super-resolved output (bottom) using C2-Matching, showing restored textures and sharp edges.

Conclusion

$C^2$-Matching effectively bridges the transformation and resolution gaps that have historically limited Ref-SR performance. By combining contrastive learning with correlation distillation, the framework provides a robust solution for both image and video enhancement. While computationally more intensive than flow-based methods, its ability to handle "in-the-wild" reference images marks a significant step toward practical, high-fidelity super-resolution.

Summary updated

BibTeX
@article{jiang2023reference,
  author = {Yuming Jiang and Kelvin C. K. Chan and Xintao Wang and Chen Change Loy and Ziwei Liu},
  title = {{Reference-based Image and Video Super-Resolution via C2-Matching}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3231089},
}
Figure