Reinforced Domain Selection for Continuous Domain Adaptation

Hanbing Liu, Huaze Tang, Yanru Wu, Yang Li, Xiao-Ping Zhang

Abstract

Continuous Domain Adaptation (CDA) bridges the distribution gap between source and target domains through intermediate datasets. Traditional CDA methods often rely on domain metadata to order these intermediate steps. This paper introduces a framework that leverages Reinforcement Learning (RL) and feature disentanglement to automatically select optimal "stepping stone" domains without metadata. By utilizing an unsupervised reward mechanism based on latent domain embeddings, the model learns an efficient transfer path. Evaluations on Rotated MNIST and medical imaging (ADNI) datasets show that this approach significantly outperforms existing methods, achieving high accuracy by filtering redundant domains and maintaining robust domain-invariant features.

Introduction

Domain shift occurs when a model trained on a source distribution fails to generalize to a target distribution due to environmental or stylistic differences. Continuous Domain Adaptation (CDA) addresses this by using intermediate domains to transition the model gradually. However, identifying the most effective sequence of intermediate domains is a complex combinatorial challenge, especially when explicit metadata (like timestamps or angles) is unavailable. Selecting a path that is too sparse leads to negative transfer, while an overly dense path increases computational cost and error accumulation.

Methodology

The proposed framework addresses domain selection through two primary components: feature disentanglement and RL-based path selection.

Feature Disentanglement

The model decomposes input data into domain-invariant features (task-relevant information) and domain-specific features (style or shift information). To ensure these components are independent, the authors employ Mutual Information Neural Estimation (MINE). This disentanglement provides a compressed, low-dimensional representation of the domain "style," which serves as an efficient state space for the RL agent.

RL-Based Path Selection

Domain selection is modeled as a Reinforcement Learning problem where an agent decides whether to include or skip an intermediate domain.

  • State: Represented by the domain-specific embeddings of the current, previous, and target domains.
  • Action: A binary decision to include the candidate domain in the transfer path.
  • Unsupervised Reward: The agent is rewarded based on the Wasserstein distance between domain embeddings. The goal is to select domains that effectively reduce the distance to the target while penalizing redundancy.
Architecture of the proposed framework, showing the feature extractor, disentanglement networks, and the RL-based policy generator.

Experimental Results and Insights

The framework was evaluated on the Rotated MNIST benchmark (11 domains with varying rotation angles) and the ADNI medical dataset (MRI images across different age groups and disease stages).

Performance and Ablation

The model achieved 93.4% accuracy on Rotated MNIST and 90.5% on ADNI, outperforming state-of-the-art CDA methods. An ablation study highlighted the necessity of the combined approach: removing the RL policy and feature disentanglement caused Rotated MNIST accuracy to drop to 50.8%.

Visualizing Disentanglement

The effectiveness of the disentanglement is visible through visualizations of the latent space. While domain-specific features remain clustered by their respective domains, the domain-invariant features overlap significantly, indicating that the model successfully learned a representation that transcends domain shifts.

3D visualization of domain-specific versus domain-invariant features, demonstrating successful disentanglement and overlap of invariant features across domains.

Discussion

A key insight is the model's ability to find a "Goldilocks" path. On Rotated MNIST, the RL agent learned to skip redundant domains (e.g., jumping from 18° to 54°), which reduces the risk of error accumulation inherent in long adaptation chains.

Future work may address the computational overhead associated with MINE and RL training. Additionally, exploring the model's performance in "open-set" scenarios—where intermediate domains might contain classes not present in the source—remains an open research direction.

Summary updated

Abstract

Continuous Domain Adaptation (CDA) effectively bridges significant domain shifts by progressively adapting from the source domain through intermediate domains to the target domain. However, selecting intermediate domains without explicit metadata remains a substantial challenge that has not been extensively explored in existing studies. To tackle this issue, we propose a novel framework that combines reinforcement learning with feature disentanglement to conduct domain path selection in an unsupervised CDA setting. Our approach introduces an innovative unsupervised reward mechanism that leverages the distances between latent domain embeddings to facilitate the identification of optimal transfer paths. Furthermore, by disentangling features, our method facilitates the calculation of unsupervised rewards using domain-specific features and promotes domain adaptation by aligning domain-invariant features. This integrated strategy is designed to simultaneously optimize transfer paths and target task performance, enhancing the effectiveness of domain adaptation processes. Extensive empirical evaluations on datasets such as Rotated MNIST and ADNI demonstrate substantial improvements in prediction accuracy and domain selection efficiency, establishing our method's superiority over traditional CDA approaches.
BibTeX
@inproceedings{liu2025reinforced,
  author = {Hanbing Liu and Huaze Tang and Yanru Wu and Yang Li and Xiao-Ping Zhang},
  title = {{Reinforced Domain Selection for Continuous Domain Adaptation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
}
Figure