Abstract
Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains. Recently, pre-trained vision-language models (VLMs) have demonstrated promising zero-shot performance by leveraging semantic information to facilitate target tasks. By aligning vision and text embeddings, VLMs have shown notable success in bridging domain gaps. However, inherent differences naturally exist between modalities, which is known as modality gap. Our findings reveal that direct UDA with the presence of modality gap only transfers modality-invariant knowledge, leading to suboptimal target performance. To address this limitation, we propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components. During training, different modality components are disentangled from VLM features then handled separately in a unified manner. At test time, modality-adaptive ensemble weights are automatically determined to maximize the synergy of different components. To evaluate instance-level modality characteristics, we design a modality discrepancy metric to categorize samples into modality-invariant, modality-specific, and uncertain ones. The modality-invariant samples are exploited to facilitate cross-modal alignment, while uncertain ones are annotated to enhance model capabilities. Building upon prompt tuning techniques, our methods achieve up to 9% performance gain with 9 times of computational efficiencies. Extensive experiments and analysis across various backbones, baselines, datasets and adaptation settings demonstrate the efficacy of our design.
Introduction
Unsupervised Domain Adaptation (UDA) is the process of adapting a model trained on a labeled "source" domain (e.g., synthetic images) to an unlabeled "target" domain (e.g., real-world photos). Recently, Vision-Language Models (VLMs) like CLIP have become central to UDA because of their extensive pre-trained knowledge of text-image alignment.
However, a persistent hurdle remains: the modality gap. Even in VLMs, vision and text features are distributed differently. Standard UDA methods often attempt to collapse this gap or ignore it, leading to "modality-invariant" learning that discards useful, modality-specific details. For instance, a vision classifier might identify a "chair" based on shape, while a text classifier might better understand a "file cabinet" through semantic context. Forcing these modalities to be identical results in a loss of such nuance.
Methodology
The proposed UniMoS++ framework explicitly disentangles vision and language features within VLMs to preserve unique information. It consists of three main pillars:
- Feature Disentanglement: A Modality Separation Network splits CLIP’s vision features into two distinct components: VAC (Vision-Associated Components) and LAC (Language-Associated Components). An orthogonal regularization loss ensures these components capture non-redundant information.
- Modality Discrepancy Metric (MDI): This metric categorizes target data into three buckets:
- Modality-Invariant (MI): High confidence agreement between vision and text.
- Modality-Specific (MS): Disagreement where one modality is more reliable.
- Uncertain (UN): Both modalities are confused; these are prioritized for human labeling in Active Domain Adaptation (ADA) settings.
- Modality-adaptive Ensemble (MaE): Unlike standard models that use fixed 50/50 weights, MaE dynamically calculates an optimal weight ($w^*$) by analyzing modality biases across the dataset, favoring the more reliable modality for specific tasks.
Crucially, the CLIP backbone remains frozen. The framework only trains lightweight linear layers and prompts, ensuring high computational efficiency.
Experimental Results
UniMoS++ was evaluated across several mainstream benchmarks, including OfficeHome, VisDA-2017, DomainNet (0.6 million images), and WILDS for out-of-distribution generalization.
- Performance: UniMoS++ outperformed previous SOTA methods on DomainNet and OfficeHome, achieving up to a 9% performance gain in active learning settings.
- Efficiency: The method is 9 times more computationally efficient than full fine-tuning methods like PADCLIP, as it utilizes frozen encoders and lightweight adapters.
- Active Learning: The MDI metric proved significantly more effective for sample selection than standard energy-based or random methods.
Conclusion
The core insight of UniMoS++ is that the modality gap is a feature, not a bug. By protecting modality-specific information rather than forcing alignment, the model retains "private" knowledge that aids generalization. While the method faces challenges in extreme synthetic-to-real gaps (VisDA) or highly specialized domains (medical imaging in WILDS) where the base VLM lacks prior knowledge, it establishes a new standard for efficient and effective domain adaptation. Future work will explore adapting to expert domains entirely unseen by the base VLM.
Summary updated