Adversarial Reinforcement Learning for Unsupervised Domain Adaptation

Youshan Zhang, Hui Ye, Brian D. Davison

Introduction

Machine learning models frequently encounter domain shift, where the data distribution at test time (target domain) differs from the training data (source domain). Unsupervised Domain Adaptation (UDA) addresses this by aligning the distributions of a labeled source domain and an unlabeled target domain. While most UDA methods utilize a single, fixed pre-trained model for feature extraction, this paper argues that a "one-size-fits-all" approach is suboptimal. Instead, it introduces Adversarial Reinforcement Learning (ARL), a framework that automatically selects the most compatible feature extractors for specific domain pairs.

The ARL Framework

The ARL framework operates in two interactive stages: feature selection via reinforcement learning and adversarial distribution alignment.

Reinforcement Learning Feature Selector

To avoid the computational cost of an exhaustive search across all possible model combinations, the authors employ a Q-learning agent. This agent navigates a library of 16 pre-trained ImageNet models (including ResNet, VGG, and Inception variants) to find the optimal backbone for both the source and target domains.

Visualization of the feature space for the library of 16 pre-trained models used in the ARL framework.

The agent is guided by a novel Deep Correlation Reward. For the unlabeled target domain, this reward uses cosine similarity to identify "neighbors" in the feature space. If the model's prediction for an image aligns with the labels of its nearest neighbors, the agent receives a positive reward, effectively providing unsupervised feedback.

Adversarial Distribution Alignment

Once the optimal feature pairs are selected, the system enters an adversarial training phase. A domain discriminator attempts to distinguish between source and target features, while the feature extractor learns to generate domain-invariant representations. This is further refined using Manifold Embedded Distribution Alignment (MEDA) to ensure consistency across both marginal and conditional distributions.

Key Insights and Results

A primary finding of this research is the effectiveness of heterogeneous backbones. The best performance often results from using different pre-trained models for the source and target domains (e.g., Xception for source and NasNetLarge for target), which can significantly reduce the initial distance between domains.

Distribution of features across different models and domains, illustrating the complexity of finding compatible backbone pairs.

ARL was evaluated on three standard benchmarks:

  • Office-Home: Achieved 75.6% average accuracy (surpassing the previous SOTA of 73.5%).
  • Office-31: Reached 91.8% average accuracy.
  • Office + Caltech-10: Achieved 97.7% average accuracy.

Conclusion

By treating feature selection as a search problem solved through reinforcement learning, ARL demonstrates that the compatibility between feature spaces is more critical than the raw accuracy of a single extractor. While currently utilizing fixed features to maintain efficiency, future iterations may explore the integration of fine-tuning or scaling the library to include newer architectures like Vision Transformers.

Summary updated

Abstract

Transferring knowledge from an existing labeled domain to a new domain often suffers from domain shift in which performance degrades because of differences between the domains. Domain adaptation has been a prominent method to mitigate such a problem. There have been many pretrained neural networks for feature extraction. However, little work discusses how to select the best feature instances across different pre-trained models for both the source and target domain. We propose a novel approach to select features by employing reinforcement learning, which learns to select the most relevant features across two domains. Specifically, in this framework, we employ Q-learning to learn policies for an agent to make feature selection decisions by approximating the action-value function. After selecting the best features, we propose an adversarial distribution alignment learning to improve the prediction results. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art method
BibTeX
@inproceedings{zhang2021adversarial,
  author = {Youshan Zhang and Hui Ye and Brian D. Davison},
  title = {{Adversarial Reinforcement Learning for Unsupervised Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00068},
}
Figure