TL;DR
This paper introduces C²-Matching, a novel approach for Reference-Based Super-Resolution (Ref-SR) that explicitly tackles the challenges of transformation and resolution gaps between low-resolution (LR) input images and high-resolution (HR) reference images. By proposing a contrastive correspondence network and teacher-student correlation distillation, C²-Matching achieves more robust and accurate texture transfer, leading to significant improvements in both image and video super-resolution tasks.
What Problem Does This Paper Address?
Traditional super-resolution (SR) methods aim to enhance the resolution of a single low-resolution (LR) image. Reference-Based Super-Resolution (Ref-SR) takes this a step further by leveraging an additional high-resolution (HR) reference image to guide the super-resolution process, allowing for more detailed texture transfer. However, existing Ref-SR methods often struggle with two main challenges:
- Transformation Gap: Differences in scale, rotation, and other geometric transformations between the LR input and HR reference images make it difficult to find accurate correspondences.
- Resolution Gap: The inherent difference in resolution between the LR input and HR reference images, coupled with potential information loss during downsampling, further complicates the matching process, especially for fine textures.
These challenges lead to inaccurate texture transfer and suboptimal super-resolution results.
Methods
The proposed C²-Matching method addresses these challenges through a two-pronged approach:
- Contrastive Correspondence Network: This network is designed to learn correspondences that are robust to transformations like scale and rotation. It achieves this by:
- Augmented Views: Training on augmented versions of the input image to simulate transformation variations.
- Triplet Margin Loss: Minimizing the distance between feature descriptors of corresponding points (positive pairs) and maximizing the distance between non-corresponding points (negative pairs). This encourages the network to learn discriminative features that are invariant to transformations.
- Teacher-Student Correlation Distillation: To bridge the resolution gap and improve matching on textured regions, a knowledge distillation approach is employed:
- Teacher Network: A contrastive correspondence network trained on HR-HR matching tasks. This "teacher" is better at handling complex textures due to the higher quality input.
- Student Network: The LR-HR matching network (the one used for the actual Ref-SR task).
- Distillation: The knowledge from the teacher network (specifically, its correlation volumes) is distilled to guide the student network. This helps the student network learn more accurate correspondences for textured regions, which are typically harder to match.
Additionally, a Dynamic Aggregation Module is introduced to fuse the aggregated reference features. This module uses the correspondences computed by the C²-Matching network and learned offsets to dynamically aggregate information around the correspondence points, handling potential misalignments more effectively than fixed-size patch cropping.
Datasets
The paper uses several datasets for training and evaluation:
- CUFED5: A standard benchmark dataset for Ref-SR, used for training and evaluation.
- SUN80, Urban100, Manga109: Datasets used for evaluating the generalizability of the image super-resolution (SR) model.
- Webly-Referenced SR (WR-SR) Dataset: A newly introduced dataset designed to mimic real-world scenarios where users search for reference images online. It contains 80 image pairs collected by searching for visually similar images on Google Image. This dataset offers a more diverse category distribution than CUFED5, with more animal, building, and landmark images.
- REDS dataset: Used for training and evaluating the video super-resolution (VSR) model.
- Vid4 dataset: Another dataset used for evaluating VSR performance.
Key Insights and Ideas
- Explicitly addressing transformation and resolution gaps is crucial: The paper strongly advocates for tackling these two fundamental challenges in Ref-SR, rather than relying solely on implicit matching.
- Contrastive learning for robust feature extraction: Using contrastive learning with augmented views is an effective way to learn features that are invariant to geometric transformations, leading to more reliable correspondences.
- Knowledge distillation for challenging domains: Teacher-student distillation is a powerful technique not just for model compression but also for transferring specialized matching capabilities (like handling fine textures) from a high-quality domain (HR-HR matching) to a more challenging one (LR-HR matching).
- Dynamic aggregation for better texture fusion: Moving beyond fixed patch-based aggregation to a dynamic approach that learns offsets and considers learned correspondences leads to more accurate and flexible texture transfer.
- Real-world relevance of WR-SR: The introduction of the WR-SR dataset highlights the need for benchmarks that better reflect practical usage scenarios, where reference images are not perfectly curated.
Key Results
- Image Super-Resolution (Ref-SR): C²-Matching achieves state-of-the-art results on the CUFED5 benchmark, outperforming existing methods by up to 0.7dB. It also demonstrates strong performance and generalizability on SUN80, Manga109, and the newly introduced WR-SR dataset.
- Video Super-Resolution (Ref-VSR): By integrating C²-Matching into existing VSR pipelines (like BasicVSR), the performance is significantly boosted. The paper shows improvements over state-of-the-art VSR methods and other Ref-VSR approaches.
- Robustness: C²-Matching exhibits superior robustness to scale and rotation transformations compared to methods like SRNTT, as evidenced by lower average end-to-point errors in matching.
- Qualitative Results: Visual comparisons show that C²-Matching recovers finer details and sharper edges, producing more realistic textures, especially in challenging scenarios. A user study indicated that over 80% of users preferred the results generated by C²-Matching.
- Efficiency: C²-Matching-light, a smaller version of the model, achieves comparable performance to SRNTT with a similar number of parameters but significantly better PSNR.
Limitations and Future Work
- Inference Time: While C²-Matching achieves superior results, its inference time is longer than methods relying solely on optical flow for alignment. The authors suggest that mechanisms for accelerating similarity matching, such as PatchMatch or MASA-SR, could be applied in future work.
- Irrelevant Reference Images: The paper notes that if an irrelevant reference image is provided, the proposed C²-Matching degenerates to a vanilla video SR method (like BasicVSR). This suggests that while C²-Matching is robust to transformations, its effectiveness is still tied to the relevance of the reference image.
- Further improvements in Ref-VSR: The authors suggest that C²-Matching can serve as a strong baseline and component for other tasks, such as capturing long-term correspondences in VSR, and that further improvements in Ref-SR can be achieved for scenarios with extremely large differences between input and reference images.
Summary updated