Abstract
Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. Meanwhile, current vision-language models exhibit remarkable zero-shot prediction capabilities. In this work, we combine knowledge gained through UDA with the inherent knowledge of vision-language models. We introduce a strong-weak guidance learning scheme that employs zero-shot predictions to help align the source and target dataset. For the strong guidance, we expand the source dataset with the most confident samples of the target dataset. Additionally, we employ a knowledge distillation loss as weak guidance. The strong guidance uses hard labels but is only applied to the most confident predictions from the target dataset. Conversely, the weak guidance is employed to the whole dataset but uses soft labels. The weak guidance is implemented as a knowledge distillation loss with (shifted) zero-shot predictions. We show that our method complements and benefits from prompt adaptation techniques for vision-language models. We conduct experiments and ablation studies on three benchmarks (OfficeHome, VisDA, and DomainNet), outperforming state-of-the-art methods. Our ablation studies further demonstrate the contributions of different components of our algorithm.
Introduction
The challenge of Unsupervised Domain Adaptation (UDA) lies in training a model on a labeled "source" dataset such that it generalizes effectively to an unlabeled "target" dataset. Traditional UDA approaches typically focus on aligning feature distributions between these domains to minimize the domain shift. However, the advent of large-scale Vision-Language (VL) models, such as CLIP, has introduced a new paradigm: zero-shot classification. These models possess a broad "world view" from their extensive pre-training, allowing them to perform well on target domains without any specific training.
Despite their success, UDA and zero-shot models have largely been treated as separate research paths. This paper addresses a fundamental question: How can we effectively combine the specific, labeled knowledge of a source dataset with the broad, inherent knowledge of a pre-trained VL model?
Methodology: Strong-Weak Guidance
The authors propose a dual-guidance learning scheme, termed Strong-Weak Guidance (SWG), integrated with standard UDA methods like CDAN. Unlike approaches that focus solely on prompt engineering, SWG adapts the visual encoder itself to the target domain.
1. Strong Guidance: Source Domain Expansion
The "strong" component of the guidance involves identifying target samples where the VL model (CLIP) exhibits high confidence. These samples are assigned "hard" pseudo-labels and appended to the source dataset. By treating these certain target images as additional training data, the model directly incorporates target-domain information into the supervised learning phase.
2. Weak Guidance: Knowledge Distillation
The "weak" guidance is applied to the entire target dataset. It utilizes CLIP’s zero-shot predictions as "soft" labels for knowledge distillation. To enhance the quality of this signal, the authors introduce Distribution Accentuation. Raw CLIP probabilities can often be overly "flat" or uncertain. By calculating a temperature parameter ($T$) to sharpen these distributions—aiming for an average winning probability of approximately 0.9—the "teacher" signal becomes significantly clearer for the student model.
3. Iterative Refinement and Architecture Adjustments
The framework is further refined in a "V2" version, which uses an iterative approach. The second training run utilizes a mixture of initial zero-shot predictions and predictions from the first run to improve pseudo-label accuracy. Additionally, for CNN-based backbones like ResNet-50, the authors adjust Batch Normalization layers to better handle the distribution shift between CLIP's pre-training data and UDA benchmarks.
Experimental Results
The SWG framework was evaluated on three major benchmarks: OfficeHome, VisDA-2017, and DomainNet.
Performance on OfficeHome
On the OfficeHome dataset, which features 65 categories across four domains, SWG demonstrated substantial improvements. Using a ResNet-50 backbone, SWG V2 achieved an average accuracy of 80.3%, outperforming both standard CLIP zero-shot (approx. 70%) and traditional UDA methods like CDAN (65.8%). When combined with prompt tuning techniques such as DAPL, performance reached 84.3%.
Large-Scale Benchmarks
In the VisDA-2017 synthetic-to-real challenge, the method achieved 90.2% with ResNet and 91.5% with ViT, surpassing specialized transformer-based UDA models. On the massive DomainNet dataset, SWG showed consistent scalability and improvement across diverse domain shifts.
Discussion and Conclusion
The primary insight of this work is the complementary nature of UDA and VL models. While UDA excels at learning domain-specific features from labeled source data, CLIP provides a robust global context that prevents the model from "forgetting" general concepts during adaptation. The study also highlights that adapting the visual encoder is not only effective but also complementary to prompt tuning; using both in tandem yields the highest performance.
While the iterative V2 approach introduces some computational overhead, the gains in accuracy justify the cost for many applications. Future research may focus on automating the balance between strong and weak guidance and exploring more complex prompt engineering synergies.
Summary updated