Unified modality separation: A vision-language framework for unsupervised domain adaptation

Xinyao Li, Jingjing Li, Zhekai Du, Lei Zhu, Heng Tao Shen

Abstract

Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains. Recently, pre-trained vision-language models (VLMs) have demonstrated promising zero-shot performance by leveraging semantic information to facilitate target tasks. By aligning vision and text embeddings, VLMs have shown notable success in bridging domain gaps. However, inherent differences naturally exist between modalities, which is known as modality gap. Our findings reveal that direct UDA with the presence of modality gap only transfers modality-invariant knowledge, leading to suboptimal target performance. To address this limitation, we propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components. During training, different modality components are disentangled from VLM features then handled separately in a unified manner. At test time, modality-adaptive ensemble weights are automatically determined to maximize the synergy of different components. To evaluate instance-level modality characteristics, we design a modality discrepancy metric to categorize samples into modality-invariant, modality-specific, and uncertain ones. The modality-invariant samples are exploited to facilitate cross-modal alignment, while uncertain ones are annotated to enhance model capabilities. Building upon prompt tuning techniques, our methods achieve up to 9% performance gain with 9 times of computational efficiencies. Extensive experiments and analysis across various backbones, baselines, datasets and adaptation settings demonstrate the efficacy of our design.

Introduction

Unsupervised Domain Adaptation (UDA) is the process of adapting a model trained on a labeled "source" domain (e.g., synthetic images) to an unlabeled "target" domain (e.g., real-world photos). Recently, Vision-Language Models (VLMs) like CLIP have become central to UDA because of their extensive pre-trained knowledge of text-image alignment.

However, a persistent hurdle remains: the modality gap. Even in VLMs, vision and text features are distributed differently. Standard UDA methods often attempt to collapse this gap or ignore it, leading to "modality-invariant" learning that discards useful, modality-specific details. For instance, a vision classifier might identify a "chair" based on shape, while a text classifier might better understand a "file cabinet" through semantic context. Forcing these modalities to be identical results in a loss of such nuance.

Comparison of UDA approaches and the proposed modality-aware adaptation.

Methodology

The proposed UniMoS++ framework explicitly disentangles vision and language features within VLMs to preserve unique information. It consists of three main pillars:

  1. Feature Disentanglement: A Modality Separation Network splits CLIP’s vision features into two distinct components: VAC (Vision-Associated Components) and LAC (Language-Associated Components). An orthogonal regularization loss ensures these components capture non-redundant information.
  2. Modality Discrepancy Metric (MDI): This metric categorizes target data into three buckets:
  3. Modality-Invariant (MI): High confidence agreement between vision and text.
  4. Modality-Specific (MS): Disagreement where one modality is more reliable.
  5. Uncertain (UN): Both modalities are confused; these are prioritized for human labeling in Active Domain Adaptation (ADA) settings.
  6. Modality-adaptive Ensemble (MaE): Unlike standard models that use fixed 50/50 weights, MaE dynamically calculates an optimal weight ($w^*$) by analyzing modality biases across the dataset, favoring the more reliable modality for specific tasks.
Visualization of modality separation and its impact on classification outcomes.

Crucially, the CLIP backbone remains frozen. The framework only trains lightweight linear layers and prompts, ensuring high computational efficiency.

Experimental Results

UniMoS++ was evaluated across several mainstream benchmarks, including OfficeHome, VisDA-2017, DomainNet (0.6 million images), and WILDS for out-of-distribution generalization.

  • Performance: UniMoS++ outperformed previous SOTA methods on DomainNet and OfficeHome, achieving up to a 9% performance gain in active learning settings.
  • Efficiency: The method is 9 times more computationally efficient than full fine-tuning methods like PADCLIP, as it utilizes frozen encoders and lightweight adapters.
  • Active Learning: The MDI metric proved significantly more effective for sample selection than standard energy-based or random methods.

Conclusion

The core insight of UniMoS++ is that the modality gap is a feature, not a bug. By protecting modality-specific information rather than forcing alignment, the model retains "private" knowledge that aids generalization. While the method faces challenges in extreme synthetic-to-real gaps (VisDA) or highly specialized domains (medical imaging in WILDS) where the base VLM lacks prior knowledge, it establishes a new standard for efficient and effective domain adaptation. Future work will explore adapting to expert domains entirely unseen by the base VLM.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains. Recently, pre-trained vision-language models (VLMs) have demonstrated promising zero-shot performance by leveraging semantic information to facilitate target tasks. By aligning vision and text embeddings, VLMs have shown notable success in bridging domain gaps. However, inherent differences naturally exist between modalities, which is known as modality gap. Our findings reveal that direct UDA with the presence of modality gap only transfers modality-invariant knowledge, leading to suboptimal target performance. To address this limitation, we propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components. During training, different modality components are disentangled from VLM features then handled separately in a unified manner. At test time, modality-adaptive ensemble weights are automatically determined to maximize the synergy of different components. To evaluate instance-level modality characteristics, we design a modality discrepancy metric to categorize samples into modality-invariant, modality-specific, and uncertain ones. The modality-invariant samples are exploited to facilitate cross-modal alignment, while uncertain ones are annotated to enhance model capabilities. Building upon prompt tuning techniques, our methods achieve up to 9% performance gain with 9 times of computational efficiencies. Extensive experiments and analysis across various backbones, baselines, datasets and adaptation settings demonstrate the efficacy of our design.
BibTeX
@article{li2025unified,
  author = {Xinyao Li and Jingjing Li and Zhekai Du and Lei Zhu and Heng Tao Shen},
  title = {{Unified modality separation: A vision-language framework for unsupervised domain adaptation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=EsL0r5GRyM},
}
Figure