Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation

Xinyao Li, Jingjing Li, Zhekai Du, Lei Zhu, Heng Tao Shen

TL;DR

This paper introduces UniMoS++, a framework that boosts Unsupervised Domain Adaptation (UDA) by explicitly disentangling vision and language features within models like CLIP. Instead of trying to force perfect alignment between modalities, the authors embrace the "modality gap" to preserve unique information, achieving up to a 9% performance gain with 9x the computational efficiency of fully fine-tuned models.

What Problem Does This Paper Address?

Unsupervised Domain Adaptation (UDA) is the art of taking a model trained on a labeled "source" domain (e.g., synthetic images) and making it work on an unlabeled "target" domain (e.g., real-world photos).

Recently, Vision-Language Models (VLMs) like CLIP have become the go-to tool for this because they already "know" a lot about the world through text-image alignment. However, there is a persistent hurdle known as the modality gap: even in CLIP, vision and text features are distributed differently. Standard UDA methods often try to collapse this gap or ignore it, which leads to "modality-invariant" learning that throws away useful, modality-specific details. For example, a vision classifier might be great at identifying a "chair" based on shape, while a text classifier might better understand a "file cabinet" through semantic context. If you force them to be identical, you lose that nuance.

Methods

The authors propose a "Unified Modality Separation" framework (UniMoS++) consisting of three main pillars:

  1. Feature Disentanglement: Using a Modality Separation Network, the model splits CLIP’s vision features into two distinct components: VAC (Vision-Associated Components) and LAC (Language-Associated Components). They use an orthogonal regularization loss to ensure these two components capture different types of information.
  2. Modality Discrepancy Metric (MDI): This novel metric categorizes target data into three buckets:
  3. Modality-Invariant (MI): Both vision and text agree (high confidence).
  4. Modality-Specific (MS): The modalities disagree, but one is clearly "more right."
    • Uncertain (UN): Both are confused. In Active Domain Adaptation (ADA) settings, these are the samples sent to humans for labeling.
  5. Modality-adaptive Ensemble (MaE): At test time, most models just average the vision and text scores (50/50). MaE automatically calculates an optimal weight ($w^*$) by analyzing the distribution of modality biases across the dataset, allowing the model to dynamically favor the more reliable modality for a specific task.

Crucially, the CLIP backbone remains frozen. The framework only trains lightweight linear layers and prompts, which is why it is so computationally efficient.

Datasets

The authors evaluated UniMoS++ across several mainstream benchmarks:

  • OfficeHome: 4 domains (Art, Clipart, Product, Realworld) with 65 categories.
  • VisDA-2017: A large-scale synthetic-to-real challenge.
  • DomainNet: The most challenging benchmark, featuring 0.6 million images across 345 categories.
  • Mini-DomainNet: A subset for faster iteration.
  • WILDS: Used for testing "out-of-distribution" generalization in real-world scenarios (e.g., medical imaging).

Key Insights and Ideas

  • Separation is Better than Forced Alignment: The "aha moment" here is that the modality gap isn't a bug; it's a feature. By protecting modality-specific information rather than trying to bridge the gap, the model retains "private" knowledge that helps it generalize to new domains.
  • The "Modality Collapse" Trap: If you only train on samples where vision and text agree, the model becomes redundant. The MDI metric helps identify where the modalities disagree, which is often where the most valuable learning happens.
  • Efficiency via Freezing: By keeping the massive VLM frozen and only tuning "adapters" (linear layers), the authors prove you don't need massive GPU clusters to achieve State-of-the-Art (SOTA) results in domain adaptation.

Key Results

  • Performance: UniMoS++ outperformed previous SOTA methods on DomainNet and OfficeHome. In some active learning settings, it achieved a 9% performance gain.
  • Efficiency: The method is 9 times more computationally efficient than full fine-tuning methods like PADCLIP, thanks to its use of frozen encoders and linear layers.
  • Versatility: The framework works across multiple settings, including standard UDA, Multi-Source DA (MSDA), and Active DA (ADA).
  • Active Learning: Using the MDI metric to select samples for human labeling proved significantly more effective than standard energy-based or random selection methods.

Limitations and Future Work

  • Synthetic-to-Real Gaps: On the VisDA dataset, UniMoS++ was slightly behind methods that fully fine-tune CLIP parameters. The authors note that the massive domain gap between synthetic and real data sometimes requires deeper architectural changes that frozen encoders can't fully capture.
  • Expert Domains: In the WILDS benchmark (specifically medical tissue slides), the model's performance is limited by CLIP's original training data. If CLIP hasn't seen it, UniMoS++ has a harder time adapting.
  • Future Direction: The authors suggest exploring how to adapt to "completely unseen" expert domains where the base VLM has zero prior knowledge.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains. Recently, pre-trained vision-language models (VLMs) have demonstrated promising zero-shot performance by leveraging semantic information to facilitate target tasks. By aligning vision and text embeddings, VLMs have shown notable success in bridging domain gaps. However, inherent differences naturally exist between modalities, which is known as modality gap. Our findings reveal that direct UDA with the presence of modality gap only transfers modality-invariant knowledge, leading to suboptimal target performance. To address this limitation, we propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components. During training, different modality components are disentangled from VLM features then handled separately in a unified manner. At test time, modality-adaptive ensemble weights are automatically determined to maximize the synergy of different components. To evaluate instance-level modality characteristics, we design a modality discrepancy metric to categorize samples into modality-invariant, modality-specific, and uncertain ones. The modality-invariant samples are exploited to facilitate cross-modal alignment, while uncertain ones are annotated to enhance model capabilities. Building upon prompt tuning techniques, our methods achieve up to 9% performance gain with 9 times of computational efficiencies. Extensive experiments and analysis across various backbones, baselines, datasets and adaptation settings demonstrate the efficacy of our design.
BibTeX
@article{li2025unified,
  author = {Xinyao Li and Jingjing Li and Zhekai Du and Lei Zhu and Heng Tao Shen},
  title = {{Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=EsL0r5GRyM},
}

Loading related content...

Figure