SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation

Hao-Wei Yeh, Baoyao Yang, Pong C. Yuen, Tatsuya HaradaWACV 2021 · OpenReview

Abstract

Applying a trained model on a new scenario may suffer from domain shift. Unsupervised domain adaptation (UDA) has been proven to be an effective approach to solve the problem of domain shift by leveraging both data from the scenario that the model was trained on (source) and the new scenario (target). Although the source data are available for training the source model, there is no guarantee that the source data will still be available when applying UDA in the future due to emerging regulations on privacy of data. This results in the in-applicability of most existing UDA methods in the absence of source data. This paper proposes a source-data-free feature alignment (SoFA) method to address this problem by only using the trained source model and unlabeled target data. The source model is used to predict the labels for target data, and we model the generation process from predicted classes to input data to infer the latent features for alignment. Specifically, a mixture of Gaussian distributions is induced from the predicted classes as the reference distribution. The encoded target features are then aligned to the reference distribution via variational inference to extract class semantics without accessing source data. Relationship of the proposed method and the theory of domain adaptation is provided to verify the performance. Experimental results show the proposed method achieves higher or comparable accuracy compared to the existing methods in several cross-dataset classification tasks. Ablation studies are also conducted to confirm the importance of latent feature alignment to adaptation performance.

Introduction

In the world of machine learning, "domain shift" is a persistent challenge. A model trained on a specific source dataset often experiences a significant performance drop when deployed in a new target environment where lighting, angles, or backgrounds differ. Traditionally, Unsupervised Domain Adaptation (UDA) addresses this by aligning features from both source and target data.

However, modern privacy regulations like GDPR and proprietary constraints often make the original source data inaccessible at the time of adaptation. This creates a need for "Source-Data-Free UDA." A new approach titled SoFA (Source-data-free Feature Alignment) introduces a method to bridge this gap using only a pre-trained source model and unlabeled target data.

The Challenge: Source-Data-Free Adaptation

The core difficulty in source-data-free UDA is the lack of a "ground truth" distribution for alignment. Most existing techniques are "data-hungry," requiring side-by-side comparisons of source and target features to minimize the distance between them. Without the source data, models risk "forgetting" class semantics or focusing on irrelevant target-specific features.

Comparison between standard Unsupervised Domain Adaptation (UDA) and the Source-data-free UDA setting where source data is unavailable.

Methodology: The SoFA Framework

The authors propose SoFA, which centers on a novel architecture called the Latent Alignment Variational Auto-Encoder (LA-VAE). The process operates through three primary mechanisms:

  1. Inducing a Reference Distribution: Since source features are unavailable, the source model is used to predict labels for the target data. A mixture of Gaussian distributions is then "induced" from these predicted classes to act as a proxy for the missing source feature space.
  2. Feature Alignment via Variational Inference: Target data is passed through an encoder. These encoded features are then "pushed" to align with the induced Gaussian mixture, forcing the model to extract features that are semantically meaningful to the original source model.
  3. Data Reconstruction: To prevent the model from collapsing features into non-informative points, a decoder attempts to reconstruct the original target image from the latent features. This ensures the features remain rich and discriminative.
The LA-VAE framework: matching target data, aligning latent features to induced distributions, and matching source-model predictions.

To stabilize the learning process, the method utilizes a "KL annealing" schedule, gradually increasing the weight of the alignment term to prevent the model from settling into poor local minima during early training stages.

Experimental Evaluation

The researchers evaluated SoFA on standard benchmarks, including Office31 (31 object classes across web, DSLR, and webcam domains) and Office-Home (65 classes across Art, Clipart, Product, and Real-world images).

Key Results

SoFA consistently outperformed or matched existing state-of-the-art methods that operate without source data, such as SHOT or RWA:

  • Office31: Achieved an average accuracy of 74.81%, significantly higher than the "Source Only" baseline of 66.50%.
  • Office-Home: Reached 74.68% average accuracy, outperforming the next best method by nearly 2%.
  • Robustness: The method proved particularly effective in difficult "Clipart-to-Real" tasks, where the domain gap is widest.

Conclusion

SoFA demonstrates that a source model's knowledge, preserved in its predictions, can be leveraged to recreate a "ghost" of the source distribution for alignment. By combining latent feature alignment with image reconstruction, the method maintains class semantics without ever needing to revisit the original training data. This represents a significant step toward flexible, privacy-respecting AI that can adapt to new environments in the wild.

Summary updated

Abstract

Applying a trained model on a new scenario may suffer from domain shift. Unsupervised domain adaptation (UDA) has been proven to be an effective approach to solve the problem of domain shift by leveraging both data from the scenario that the model was trained on (source) and the new scenario (target). Although the source data are available for training the source model, there is no guarantee that the source data will still be available when applying UDA in the future due to emerging regulations on privacy of data. This results in the in-applicability of most existing UDA methods in the absence of source data. This paper proposes a source-data-free feature alignment (SoFA) method to address this problem by only using the trained source model and unlabeled target data. The source model is used to predict the labels for target data, and we model the generation process from predicted classes to input data to infer the latent features for alignment. Specifically, a mixture of Gaussian distributions is induced from the predicted classes as the reference distribution. The encoded target features are then aligned to the reference distribution via variational inference to extract class semantics without accessing source data. Relationship of the proposed method and the theory of domain adaptation is provided to verify the performance. Experimental results show the proposed method achieves higher or comparable accuracy compared to the existing methods in several cross-dataset classification tasks. Ablation studies are also conducted to confirm the importance of latent feature alignment to adaptation performance.
BibTeX
@inproceedings{yeh2021sofa,
  author = {Hao-Wei Yeh and Baoyao Yang and Pong C. Yuen and Tatsuya Harada},
  title = {{SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=hr_oOVPyBg2Y},
}
Figure