Adapting Without Seeing: Text-Aided Domain Adaptation for Adapting CLIP-like Models to Novel Domains

Louis Hémadou, Héléne Vorobieva, Ewa Kijak, Frédéric JurieICASSP 2025 · DOI

TL;DR

This paper introduces TADA, a novel method for adapting large vision-language models like CLIP to new domains without needing any target domain images. TADA leverages textual descriptions of the target domain to guide the adaptation of visual representations, achieving state-of-the-art results on challenging domain adaptation benchmarks.

What Problem Does This Paper Address?

Large vision-language models, such as CLIP, have shown remarkable capabilities due to their pre-training on massive image-text datasets. However, their performance can significantly degrade when applied to domains that differ from their training data, a common issue in specialized fields like industrial applications. Traditional domain adaptation methods often require target domain data (even if unlabeled), which can be scarce or unavailable. Domain generalization aims to create models robust to unseen domains without target data, but adapting existing powerful models like CLIP to specific new domains remains a challenge. This paper tackles the problem of adapting CLIP-like models to new, unseen domains when target domain images are unavailable, relying solely on textual descriptions of the target domain.

Methods

The core of TADA (Text-Aided Domain Adaptation) is to learn an "adaptation function" that transforms textual representations from the source domain to the target domain. This function is then applied to the image features extracted from the source domain, effectively shifting them into the target domain's representation space within CLIP's latent space.

Here's a breakdown of the key steps:

  1. Caption Generation: Generic captions are generated using a large language model (LLaMA3) based on a diverse set of visual concepts.
  2. Domain-Specific Caption Pairs: These generic captions are then adapted to specific source and target domains by replacing a generic word like "picture" with domain-specific textual descriptions (e.g., "photo," "cartoon"). This creates pairs of source and target captions that capture the domain shift.
  3. Adaptation Function Learning: For each target domain, an adaptation function is learned. This function takes the CLIP text embedding of a source caption and transforms it into the CLIP text embedding of its corresponding target caption. This is achieved by minimizing the L2 loss between the transformed source embedding and the target embedding. The adaptation functions are implemented as simple fully-connected neural networks.
  4. Feature Augmentation: The learned adaptation function is then applied to the image features of the source domain. This effectively "augments" the source image features to align them with the target domain's characteristics.
  5. Fine-tuning: Finally, CLIP is fine-tuned on these augmented image representations for the target domain. The authors emphasize that the visual encoder of CLIP remains frozen during fine-tuning to preserve its critical text-image alignment and reduce computational demands. This allows TADA to be compatible with various fine-tuning strategies, including linear probing and prompt learning.

Datasets

The paper evaluates TADA on three standard domain adaptation benchmarks:

  • DomainNet: A large-scale dataset comprising 345 classes and six domains (quickdraw, infograph, sketch, painting, clipart, real).
  • OfficeHome: A dataset with 65 classes and four domains (art, clipart, product, real world).
  • PACS: A dataset containing seven classes and four domains (photo, art painting, cartoon, sketch).

For training, the models are typically trained on the "photo realistic" domain of each dataset and evaluated on the remaining unseen domains.

Key Insights and Ideas

  • Leveraging Textual Descriptions for Domain Adaptation: The core insight is that rich textual descriptions of target domains can serve as a powerful signal for adapting models without requiring any target domain images. This is particularly valuable in scenarios where data is scarce or expensive.
  • CLIP's Latent Space as a Bridge: TADA exploits the inherent alignment between text and image representations in CLIP's latent space. By operating within this space, transformations guided by text can effectively influence visual representations.
  • Isometry for Adaptation Functions: Modeling adaptation functions as isometries (preserving distances and angles) is motivated by the geometric properties of CLIP's space, where semantically related concepts are close. This choice helps maintain the integrity of the learned representations.
  • Compatibility with Existing Fine-tuning Methods: TADA is designed to be a plug-and-play module that can be integrated with various existing fine-tuning strategies for CLIP, enhancing their performance on domain-shifted data.

Key Results

TADA consistently improves the performance of CLIP-based models across the evaluated datasets (DomainNet, OfficeHome, and PACS) when fine-tuned with methods like Linear Probing (LP) and CoOp. The paper reports significant accuracy gains compared to baseline methods without adaptation. For instance, on DomainNet, TADA combined with LP achieves a notable improvement. The gains are particularly pronounced on DomainNet, suggesting its effectiveness on domains with clearer verbal distinctions.

Limitations and Future Work

  • Domain Description Specificity: The effectiveness of TADA is dependent on the quality and specificity of the textual descriptions provided for the target domains. Domains with subtle or ambiguous distinctions (e.g., "product" vs. "real world" in OfficeHome) may see less benefit.
  • Subtler Domain Shifts: The authors suggest future work should focus on extending TADA to address subtler domain shifts, potentially by exploring alternative representations beyond text captions or incorporating domain-invariant learning techniques.
  • Caption Generation Refinement: Investigating methods to refine caption generation, perhaps by utilizing source images, is another promising avenue for future research.

Summary updated

BibTeX
@inproceedings{hemadou2025adapting,
  author = {Louis Hémadou and Héléne Vorobieva and Ewa Kijak and Frédéric Jurie},
  title = {{Adapting Without Seeing: Text-Aided Domain Adaptation for Adapting CLIP-like Models to Novel Domains}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889681},
}
Figure