DRIVE: Dual-Robustness via Information Variability and Entropic Consistency in Source-Free Unsupervised Domain Adaptation

Ruiqiang Xiao, Songning Lai, Yijun Yang, Jiemin Wu, Yutao Yue, Lei ZhuarXiv 2024 · OpenReview

Abstract

Adapting machine learning models to new domains without labeled data, especially when source data is inaccessible, is a critical challenge in applications like medical imaging, autonomous driving, and remote sensing. This task, known as Source-Free Unsupervised Domain Adaptation (SFUDA), involves adapting a pre-trained model to a target domain using only unlabeled target data, which can lead to issues such as overfitting, underfitting, and poor generalization due to domain discrepancies and noise. Existing SFUDA methods often rely on single-model architectures, struggling with uncertainty and variability in the target domain. To address these challenges, we propose DRIVE (Dual-Robustness through Information Variability and Entropy), a novel SFUDA framework leveraging a dual-model architecture. The two models, initialized with identical weights, work in parallel to capture diverse target domain characteristics. One model is exposed to perturbations via projection gradient descent (PGD) guided by mutual information, focusing on high-uncertainty regions. We also introduce an entropy-aware pseudo-labeling strategy that adjusts label weights based on prediction uncertainty, ensuring the model focuses on reliable data while avoiding noisy regions. The adaptation process has two stages: the first aligns the models on stable features using a mutual information consistency loss, and the second dynamically adjusts the perturbation level based on the loss from the first stage, encouraging the model to explore a broader range of the target domain while preserving existing performance. This enhances generalization capabilities and robustness against interference. Evaluations on standard SFUDA benchmarks show that DRIVE consistently outperforms previous methods, delivering improved adaptation accuracy and stability across complex target domains.

The Challenge of Source-Free Domain Adaptation

In many real-world scenarios—like medical imaging or autonomous driving—high-performing pre-trained models are often available, but the original training data is not. Privacy laws, proprietary restrictions, or massive storage costs frequently prevent access to source data. When attempting to adapt these models to a new target domain using only unlabeled data, models often suffer from "brittleness," overfitting to noise or failing to generalize due to significant domain shifts.

Existing methods typically rely on single-model architectures. These architectures struggle to handle the inherent uncertainty and variability of a new environment, often collapsing when faced with noisy target data.

The DRIVE Framework

To overcome these limitations, we introduce DRIVE (Dual-Robustness through Information Variability and Entropy). DRIVE utilizes a two-stage, dual-model approach designed to balance feature consistency with exploratory divergence.

Overview of the two-stage adaptation process

Methodology Overview

  1. Dual-Model Architecture: The framework employs two identical models working in parallel. While one model processes "clean" target data, the other is exposed to perturbations generated via Projected Gradient Descent (PGD). This setup forces the system to identify features that are both stable (consistent across models) and capable of exploring the target domain's full breadth.
  2. Entropy-Aware Pseudo-Labeling: Rather than treating all unlabeled data equally, DRIVE calculates the "uncertainty" (entropy) of its predictions. It assigns higher weights to reliable, low-entropy regions while avoiding over-reliance on noisy or uncertain data.
  3. Two-Stage Adaptation:
    • Stage 1 (Customization): This stage aligns a vision-language (ViL) model (such as CLIP) with the target domain using a mutual information consistency loss.
  4. Stage 2 (Knowledge Adaptation): The target model learns from the customized ViL model. The level of perturbation in this stage is dynamically adjusted based on the performance of Stage 1; if the model shows high consistency in the first stage, it is encouraged to explore more aggressively in the second.

Experimental Evaluation

The DRIVE framework was evaluated on three standard benchmarks of increasing complexity:

  • Office-31: A small-scale dataset with 31 categories across three domains (Amazon, DSLR, Webcam).
  • Office-Home: A medium-scale dataset with 65 categories across four domains (Art, Clipart, Product, Real-world).
  • DomainNet-126: A large-scale, highly diverse subset featuring 126 classes.

Key Results

DRIVE consistently outperformed previous state-of-the-art (SOTA) methods across all benchmarks. On the Office-31 dataset, DRIVE achieved a mean accuracy of 92.7%, notably reaching 97.4% on the challenging Amazon $\rightarrow$ DSLR task.

Performance comparison on Office-31

In larger-scale tests, DRIVE reached a mean accuracy of 83.6% on Office-Home and 80.6% on DomainNet-126. Qualitative analysis using Grad-CAM visualizations confirmed that DRIVE focuses more accurately on relevant object features compared to baseline models, which are often distracted by background noise.

Conclusion

By forcing two models to agree on stable features while one is actively perturbed, DRIVE learns a more robust representation than single-model approaches. The feedback loop between alignment and exploration prevents the model from diverging into noise, ensuring stability even in complex target domains. Future work may extend this dual-robustness framework to other tasks such as object detection and semantic segmentation in source-free environments.

Summary updated

Abstract

Adapting machine learning models to new domains without labeled data, especially when source data is inaccessible, is a critical challenge in applications like medical imaging, autonomous driving, and remote sensing. This task, known as Source-Free Unsupervised Domain Adaptation (SFUDA), involves adapting a pre-trained model to a target domain using only unlabeled target data, which can lead to issues such as overfitting, underfitting, and poor generalization due to domain discrepancies and noise. Existing SFUDA methods often rely on single-model architectures, struggling with uncertainty and variability in the target domain. To address these challenges, we propose DRIVE (Dual-Robustness through Information Variability and Entropy), a novel SFUDA framework leveraging a dual-model architecture. The two models, initialized with identical weights, work in parallel to capture diverse target domain characteristics. One model is exposed to perturbations via projection gradient descent (PGD) guided by mutual information, focusing on high-uncertainty regions. We also introduce an entropy-aware pseudo-labeling strategy that adjusts label weights based on prediction uncertainty, ensuring the model focuses on reliable data while avoiding noisy regions. The adaptation process has two stages: the first aligns the models on stable features using a mutual information consistency loss, and the second dynamically adjusts the perturbation level based on the loss from the first stage, encouraging the model to explore a broader range of the target domain while preserving existing performance. This enhances generalization capabilities and robustness against interference. Evaluations on standard SFUDA benchmarks show that DRIVE consistently outperforms previous methods, delivering improved adaptation accuracy and stability across complex target domains.
BibTeX
@misc{xiao2024drive,
  author = {Ruiqiang Xiao and Songning Lai and Yijun Yang and Jiemin Wu and Yutao Yue and Lei Zhu},
  title = {{DRIVE: Dual-Robustness via Information Variability and Entropic Consistency in Source-Free Unsupervised Domain Adaptation}},
  year = {2024},
  eprint = {2411.15976},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15976},
}
Figure