CLIP-aware Domain-Adaptive Super-Resolution

Zhengyang Lu, Qian Xia, Weifan Wang, Feng Wang

Abstract

This work introduces CLIP-aware Domain-Adaptive Super-Resolution (CDASR), a novel framework that addresses the critical challenge of domain generalization in single image super-resolution. By leveraging the semantic capabilities of CLIP (Contrastive Language-Image Pre-training), CDASR achieves unprecedented performance across diverse domains and extreme scaling factors. The proposed method integrates CLIP-guided feature alignment mechanism with a meta-learning inspired few-shot adaptation strategy, enabling efficient knowledge transfer and rapid adaptation to target domains. A custom domain-adaptive module processes CLIP features alongside super-resolution features through a multi-stage transformation process, including CLIP feature processing, spatial feature generation, and feature fusion. This intricate process ensures effective incorporation of semantic information into the super-resolution pipeline. Additionally, CDASR employs a multi-component loss function that combines pixel-wise reconstruction, perceptual similarity, and semantic consistency. Extensive experiments on benchmark datasets demonstrate CDASR's superiority, particularly in challenging scenarios. On the Urban100 dataset at $\times$8 scaling, CDASR achieves a significant PSNR gain of 0.15dB over existing methods, with even larger improvements of up to 0.30dB observed at $\times$16 scaling.

Introduction

Single-image super-resolution (SISR) has advanced significantly with the advent of CNNs and Transformers. However, these models often struggle with domain shift. A model trained on high-quality natural photographs (e.g., the DIV2K dataset) frequently fails when applied to distinct domains like urban architecture or artistic illustrations (e.g., Manga).

As scaling factors reach $\times$8 or $\times$16, the "hallucination" problem becomes critical; the model must invent details rather than just sharpening edges. Without high-level semantic understanding, models often produce blurry or semantically incorrect textures. Visualizing the semantic feature space via t-SNE reveals that datasets like Manga109 exist as "islands" far from standard training sets, justifying the need for explicit domain adaptation.

t-SNE visualization of CLIP features across different datasets, illustrating the significant domain gap between natural images and specialized domains like Manga109.

Methodology

The CDASR framework introduces three primary innovations to bridge the semantic gap:

1. CLIP-Guided Feature Alignment

CDASR utilizes a frozen CLIP vision encoder to extract domain-agnostic semantic features. These features are processed through a custom module that reshapes them into a spatial format and fuses them with features from a standard SR backbone (like EDSR). This ensures the reconstruction is "aware" of the image's high-level content, such as distinguishing between a "building facade" and a "butterfly wing."

2. Meta-Learning Adaptation Strategy

To handle new domains efficiently, CDASR employs a few-shot adaptation technique inspired by meta-learning algorithms like MAML or Reptile. This allows the model to "learn how to learn," enabling it to adapt to a specific target domain using only a handful of labeled examples without destroying foundational knowledge.

3. Multi-Component Loss Function

Beyond standard pixel-wise ($L_1$) and perceptual losses, a Semantic Consistency Loss is introduced. This loss minimizes the distance between the CLIP features of the super-resolved output and the ground-truth high-resolution image, forcing the model to maintain semantic integrity.

The CDASR architecture overview, showcasing the integration of CLIP features with the SR backbone through the alignment and reconstruction modules.

Experiments and Results

The model was pre-trained on the DF2K dataset (DIV2K + Flickr2K) and evaluated on standard benchmarks including Set5, Set14, BSD100, Urban100, and Manga109. CDASR consistently outperforms state-of-the-art methods like SwinIR and EDT, especially at extreme scales:

  • Urban100: Achieved a 0.15dB PSNR gain at $\times$8 and 0.21dB at $\times$16.
  • Manga109: Showed the most dramatic improvements, with gains of up to 0.30dB PSNR at $\times$16.
  • Qualitative Quality: Visually, CDASR produces cleaner line art in manga and sharper edges in architectural scenes, avoiding the "melted" artifacts common in other high-scale SR outputs.

Discussion and Conclusion

While CDASR sets a new benchmark, it faces challenges with semantic ambiguity in complex scenes (e.g., reflections) and text reconstruction. Since CLIP is biased toward natural images, fine details of complex characters can be lost. Future work will explore specialized text modules and temporal integration for Video Super-Resolution.

Summary updated

Abstract

This work introduces CLIP-aware Domain-Adaptive Super-Resolution (CDASR), a novel framework that addresses the critical challenge of domain generalization in single image super-resolution. By leveraging the semantic capabilities of CLIP (Contrastive Language-Image Pre-training), CDASR achieves unprecedented performance across diverse domains and extreme scaling factors. The proposed method integrates CLIP-guided feature alignment mechanism with a meta-learning inspired few-shot adaptation strategy, enabling efficient knowledge transfer and rapid adaptation to target domains. A custom domain-adaptive module processes CLIP features alongside super-resolution features through a multi-stage transformation process, including CLIP feature processing, spatial feature generation, and feature fusion. This intricate process ensures effective incorporation of semantic information into the super-resolution pipeline. Additionally, CDASR employs a multi-component loss function that combines pixel-wise reconstruction, perceptual similarity, and semantic consistency. Extensive experiments on benchmark datasets demonstrate CDASR's superiority, particularly in challenging scenarios. On the Urban100 dataset at $\times$8 scaling, CDASR achieves a significant PSNR gain of 0.15dB over existing methods, with even larger improvements of up to 0.30dB observed at $\times$16 scaling.
BibTeX
@inproceedings{lu2025clip,
  author = {Zhengyang Lu and Qian Xia and Weifan Wang and Feng Wang},
  title = {{CLIP-aware Domain-Adaptive Super-Resolution}},
  booktitle = {Multimedia},
  year = {2025},
  doi = {10.1007/s00530-025-01849-8},
}
Figure