Introduction
Machine learning models frequently encounter domain shift, where the data distribution at test time (target domain) differs from the training data (source domain). Unsupervised Domain Adaptation (UDA) addresses this by aligning the distributions of a labeled source domain and an unlabeled target domain. While most UDA methods utilize a single, fixed pre-trained model for feature extraction, this paper argues that a "one-size-fits-all" approach is suboptimal. Instead, it introduces Adversarial Reinforcement Learning (ARL), a framework that automatically selects the most compatible feature extractors for specific domain pairs.
The ARL Framework
The ARL framework operates in two interactive stages: feature selection via reinforcement learning and adversarial distribution alignment.
Reinforcement Learning Feature Selector
To avoid the computational cost of an exhaustive search across all possible model combinations, the authors employ a Q-learning agent. This agent navigates a library of 16 pre-trained ImageNet models (including ResNet, VGG, and Inception variants) to find the optimal backbone for both the source and target domains.
The agent is guided by a novel Deep Correlation Reward. For the unlabeled target domain, this reward uses cosine similarity to identify "neighbors" in the feature space. If the model's prediction for an image aligns with the labels of its nearest neighbors, the agent receives a positive reward, effectively providing unsupervised feedback.
Adversarial Distribution Alignment
Once the optimal feature pairs are selected, the system enters an adversarial training phase. A domain discriminator attempts to distinguish between source and target features, while the feature extractor learns to generate domain-invariant representations. This is further refined using Manifold Embedded Distribution Alignment (MEDA) to ensure consistency across both marginal and conditional distributions.
Key Insights and Results
A primary finding of this research is the effectiveness of heterogeneous backbones. The best performance often results from using different pre-trained models for the source and target domains (e.g., Xception for source and NasNetLarge for target), which can significantly reduce the initial distance between domains.
ARL was evaluated on three standard benchmarks:
- Office-Home: Achieved 75.6% average accuracy (surpassing the previous SOTA of 73.5%).
- Office-31: Reached 91.8% average accuracy.
- Office + Caltech-10: Achieved 97.7% average accuracy.
Conclusion
By treating feature selection as a search problem solved through reinforcement learning, ARL demonstrates that the compatibility between feature spaces is more critical than the raw accuracy of a single extractor. While currently utilizing fixed features to maintain efficiency, future iterations may explore the integration of fine-tuning or scaling the library to include newer architectures like Vision Transformers.
Summary updated