Dual Moving Average Pseudo-Labeling for Source-Free Inductive Domain Adaptation

Hao Yan, Yuhong Guo

TL;DR

This paper introduces Dual Moving Average Pseudo-Labeling (DMAPL), a new framework for adapting pre-trained models to new domains without needing access to the original source data. Unlike most previous methods that "cheat" by testing on the same data they train on (transductive), DMAPL focuses on the more realistic inductive setting—ensuring the model generalizes well to completely unseen data in the target domain.

What Problem Does This Paper Address?

In the world of machine learning, Unsupervised Domain Adaptation (UDA) is the art of taking a model trained on one dataset (the source) and making it work on a different but related dataset (the target) without needing new labels. However, "Source-Free" Domain Adaptation (SFDA) adds a massive constraint: for privacy or efficiency reasons, you no longer have access to the original source data—only the pre-trained model.

The authors point out a major flaw in current SFDA research: most existing methods are evaluated in a transductive setting, where the model is tested on the same unlabeled data it used for adaptation. In the real world, we need inductive models that can adapt to a training set and then perform reliably on new, unseen "out-of-the-box" test data. Current methods often overfit the unlabeled training set, leading to poor generalization.

Methods

The authors propose DMAPL, which uses a "divide and conquer" strategy combined with a double-layered temporal smoothing mechanism:

  1. Target Training Data Splitting: The model first looks at the unlabeled target data and splits it into two groups based on the source model's confidence.
  2. Confident Subset ($\mathcal{L}$): Instances where the source model is very sure of the label. These are treated as "trusty" fixed labels to preserve the original model's knowledge.
  3. Less-Confident Subset ($\mathcal{U}$): Instances where the model is unsure. These are used for semi-supervised adaptation.
  4. Dual Moving Average (DMA): To handle the noise in the less-confident subset, the authors use two levels of moving averages:
  5. Prototypical Centroids: Instead of calculating class centers (prototypes) from scratch every batch, they maintain a moving average of class centers ($\mu_c$) to ensure the "geometry" of the feature space evolves smoothly.
  6. Soft-Label Updating: Instead of assigning a hard "one-hot" label to an image, they maintain a moving average of the predicted labels ($q_i$). This prevents the model from oscillating wildly between different class assignments during training.

By combining these, the model gradually "drifts" from the source domain toward the target domain without losing its foundational logic.

Datasets

The researchers evaluated DMAPL on two massive benchmarks:

  • DomainNet: A large-scale dataset with 345 categories across domains like clipart, painting, and real photos. They focused on 4 domains (12 tasks total).
  • VisDA2017Split: A synthetic-to-real challenge. Since the original VisDA doesn't have a standard train/test split for the target domain, the authors created VisDA2017Split, using an 8:2 ratio to ensure a proper inductive evaluation.

Key Insights and Ideas

  • Inductive is Harder: The paper reveals that many "state-of-the-art" methods perform significantly worse when they can't see the test data during training. For example, the SHOT method performs well in transductive settings but struggles more in the inductive setting compared to DMAPL.
  • The "Trusty" Anchor: By freezing the labels of the high-confidence subset, the authors create an anchor. This prevents "catastrophic forgetting" where the model adapts so much to the new noise that it forgets how to classify basic shapes or features learned from the source.
  • Slow and Steady Wins: The hyperparameter analysis shows that higher values for $\alpha$ and $\beta$ (slower updates) lead to better performance. This suggests that in domain adaptation, stability is more important than rapid convergence.

Key Results

DMAPL achieved state-of-the-art results across the board:

  • DomainNet: DMAPL reached an average accuracy of 55.4%, outperforming the previous best source-free method (SSFT-SSD at 50.4%) and even beating UDA methods like CDAN (49.9%) which actually have access to the source data.
  • VisDA2017Split: It achieved 83.0% (Macro accuracy), a significant jump over the source-only baseline of 48.9%.
  • Ablation Study: The authors proved that both the data splitting and the soft-label updating are necessary; removing either component led to a 3-7% drop in accuracy.

Limitations and Future Work

While DMAPL narrows the gap, it still falls short of the "Oracle" performance (training on fully labeled target data), which sits at 75.7% on DomainNet.

The authors suggest that future work should continue to move away from transductive "toy" settings and focus on even more realistic inductive scenarios. They also note that while the confidence threshold ($p_{th}$) is robust, finding ways to automatically tune these hyperparameters for different domains remains an open challenge.

Summary updated

Abstract

Unsupervised domain adaptation reduces the reliance on data annotation in deep learning by adapting knowledge from a source to a target domain. For privacy and efficiency concerns, source-free domain adaptation extends unsupervised domain adaptation by adapting a pre-trained source model to an unlabeled target domain without accessing the source data. However, most existing source-free domain adaptation methods to date focus on the transductive setting, where the target training set is also the testing set. In this paper, we address source-free domain adaptation in the more realistic inductive setting, where the target training and testing sets are mutually exclusive. We propose a new semi-supervised fine-tuning method named Dual Moving Average Pseudo-Labeling (DMAPL) for source-free inductive domain adaptation. We first split the unlabeled training set in the target domain into a pseudo-labeled confident subset and an unlabeled less-confident subset according to the prediction confidence scores from the pre-trained source model. Then we propose a soft-label moving-average updating strategy for the unlabeled subset based on a moving-average prototypical classifier, which gradually adapts the source model towards the target domain. Experiments show that our proposed method achieves state-of-the-art performance and outperforms previous methods by large margins.
BibTeX
@inproceedings{yan2022dual,
  author = {Hao Yan and Yuhong Guo},
  title = {{Dual Moving Average Pseudo-Labeling for Source-Free Inductive Domain Adaptation}},
  booktitle = {BMVC},
  year = {2022},
  url = {https://openreview.net/forum?id=Tkxqh8-Ll7},
}

Loading related content...

Figure