Autonomous Source Knowledge Selection in Multi-Domain Adaptation

Keqiuyin Li, Jie Lu, Hua Zuo, Guangquan Zhang

Abstract

Unsupervised multi-domain adaptation plays a key role in transfer learning by leveraging acquired rich source information from multiple source domains to solve target tasks from an unlabeled target domain. However, multiple source domains often contain much redundant or unrelated information which can harm transfer performance, especially in massive-source domain settings. It is urgent to develop effective strategies for identifying and selecting the most transferable knowledge from massive source domains to address the target task. In this paper, we propose a multi-domain adaptation method named Autonomous Source Knowledge Selection (AutoS) to autonomously select source training samples and models, enabling the prediction of target tasks using more relevant and transferable source information. The proposed method employs a density-driven selection strategy to choose source samples during training and to determine which source models should contribute to target prediction. Simultaneously, a pseudo-label enhancement module built on a pre-trained multimodal model is employed to mitigate target label noise and improve self-supervision. Experiments on real-world datasets indicate the superiority of the proposed method.

Introduction

In Unsupervised Multi-Domain Adaptation (UMDA), the goal is to use labeled data from several "source" domains (e.g., photos, sketches, and clip-art) to train a model for an unlabeled "target" domain. The primary challenge is that as the number of source domains grows, so does the amount of redundant or unrelated information. Standard methods typically attempt to align all source domains with the target, which is computationally expensive and often leads to negative transfer—where noisy source data confuses the model and degrades performance on the target task.

Motivation for autonomous source selection: traditional methods of training independent models or combining all domains are inefficient compared to pruning irrelevant sources.

Existing "distillation" methods to filter this data often require training models multiple times, making them impractical for massive-scale datasets. AutoS addresses this by dynamically pruning redundant source domains and samples using a density-driven strategy, combined with a multimodal foundation model to refine target predictions.

Methodology

The AutoS framework is designed to select the best knowledge autonomously through two primary stages:

1. Target-Driven Source Selection

Instead of training independent models for every source, AutoS uses a single shared model updated in a federated manner. It employs a density-controlled selection strategy:

  • Sample Filtering: The system calculates cluster centers for source data and determines a "radius" for each category. Source samples that fall outside a specific density threshold relative to the target domain are discarded.
  • Domain Pruning: Entire source domains are progressively removed during training if their "transferability weight" (based on sample density) falls below a certain threshold.

2. Target Model Adaptation

Once the best source knowledge is selected, the model adapts to the target domain. To bridge the remaining gap, AutoS uses a pseudo-label enhancement module powered by a frozen multimodal foundation model (like CLIP). It uses prompt tuning to align visual features with text descriptions, providing a high-quality self-supervision signal that mitigates noise in the target labels.

The complete AutoS architecture, illustrating the autonomous selection of source domains and samples (top) and the subsequent target adaptation with multimodal information fusion (bottom).

Experimental Evaluation

The method was evaluated on four standard real-world visual classification benchmarks: Office31, OfficeHome, DomainNet126, and the full DomainNet dataset (345 categories across 6 domains).

Key Results

  • Superior Accuracy: AutoS consistently outperformed state-of-the-art baselines (like CAiDA and FuzHDA). For example, on DomainNet126, AutoS achieved an average accuracy of 83.2%.
  • Resource Efficiency: Compared to the SSD baseline, AutoS reduced training time by nearly 7x on some tasks and used significantly less GPU memory.
  • Robustness: The "AutoS/sf" variant (source-free adaptation) performed remarkably well, proving the robustness of foundation-model-guided adaptation.

Discussion and Insights

  • Less is More: Removing source samples and entire domains doesn't just save time—it improves accuracy by pruning "dissimilar" knowledge that causes feature confusion.
  • Density as a Proxy for Transferability: Spatial density in a latent feature space is a reliable indicator of domain match. If target samples don't cluster well near a source's centers, that source is likely a liability.
  • The "Average Radius" Sweet Spot: Evaluation of different cluster boundaries (Average, RMS, and Maximum radius) showed that the Average Radius provided the best balance, effectively filtering noise without being overly restrictive.

Conclusion

AutoS provides a more accurate and memory-efficient adaptation process by focusing only on the most transferable knowledge. While it excels at handling distribution shifts, future work will explore label shifts and modality shifts (e.g., moving from image data to sensor data) to handle more complex "universal" adaptation scenarios where category sets might not perfectly overlap.

Summary updated

Abstract

Unsupervised multi-domain adaptation plays a key role in transfer learning by leveraging acquired rich source information from multiple source domains to solve target task from an unlabeled target domain. However, multiple source domains often contain much redundant or unrelated information which can harm transfer performance, especially when in massive-source domain settings. It is urgent to develop effective strategies for identifying and selecting the most transferable knowledge from massive source domains to address the target task. In this paper, we propose a multi-domain adaptation method named \underline{\textit{Auto}}nomous Source Knowledge \underline{\textit{S}}election (AutoS) to autonomosly select source training samples and models, enabling the prediction of target task using more relevant and transferable source information. The proposed method employs a density-driven selection strategy to choose source samples during training and to determine which source models should contribute to target prediction. Simulteneously, a pseudo-label enhancement module built on a pre-trained multimodal modal is employed to mitigate target label noise and improve self-supervision. Experiments on real-world datasets indicate the superiority of the proposed method.
BibTeX
@misc{li2025autonomous,
  author = {Keqiuyin Li and Jie Lu and Hua Zuo and Guangquan Zhang},
  title = {{Autonomous Source Knowledge Selection in Multi-Domain Adaptation}},
  year = {2025},
  eprint = {2512.14710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.14710},
}
Figure