Autonomous Source Knowledge Selection in Multi-Domain Adaptation

Keqiuyin Li, Jie Lu, Hua Zuo, Guangquan ZhangSubmission to ICLR 2026 · Withdrawn

TL;DR

When adapting models across multiple source domains, more data isn't always better: irrelevant or noisy sources often degrade transfer performance. AutoS (Autonomous Source Knowledge Selection) solves this by dynamically pruning uninformative source domains and samples using a density-driven selection metric in a federated training setup, followed by prompt-tuned self-supervision from a frozen vision-language model (CLIP). This allows models to achieve state-of-the-art multi-domain adaptation performance using only the most transferable source knowledge at a fraction of the computational and memory cost.

What Problem Does This Paper Address?

Unsupervised multi-domain adaptation (MDA) aims to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Traditional methods combine all available sources—often via simple feature alignment or linear parameter averaging.

However, in massive multi-source settings, many domains or sample clusters contain redundant or completely irrelevant information that triggers negative transfer. Prior attempts to filter source domains (such as sample and source distillation) relied on manual heuristics or required training source models twice—once to evaluate domain similarity and once for actual adaptation—making them computationally prohibitive. AutoS tackles the challenge of autonomously identifying and pruning unhelpful source knowledge on the fly during training.

Methods

AutoS operates in a two-stage pipeline:

[Massive Source Domains + Target] 
       │
       ▼ (Stage 1: Federated Source Training & Autonomous Pruning)
  • Single shared backbone across sources
  • Compute source cluster centers and radii (Rd)
  • Filter source/target confident samples based on distance thresholds
  • Compute target data density (ρ) and source importance weights (ω)
  • Prune irrelevant sources (Keep(D_s) thresholding)
       │
       ▼ (Stage 2: Target Model Adaptation)
  • Weighted aggregation of surviving source models
  • Cross-modal guidance from frozen CLIP (prompt tuning via information bottleneck loss Lex)
  • Target self-supervision (Lin)
  • Density-Controlled Autonomous Selection (Stage 1): Instead of training isolated source models, AutoS initializes a single shared model updated in a federated manner. It computes class cluster centers from classifier weights and calculates cluster radii. It forms an intermediate domain by retaining confident source samples and pulling in high-confidence target samples. Source domains are evaluated via a composite weight combining target sample assignment ratio and cluster density; domains falling below a dynamic threshold are pruned mid-training.
  • Cross-Modality Target Adaptation (Stage 2): Surviving source models are aggregated into a target model using their domain weights. To bridge remaining domain gaps without requiring raw source data matching, AutoS uses a frozen foundation model (CLIP). It optimizes learnable text prompts through a Gaussian structural causal/information-bottleneck loss ($L_{ex}$) and adapts the target model via entropy and self-supervision objectives ($L_{in}$).

Datasets

AutoS was evaluated on four standard benchmark datasets (no new benchmarks were introduced):

  • Office-31: 3 domains (Amazon, DSLR, Webcam), 31 classes.
  • Office-Home: 4 domains (Art, Clipart, Product, Real-World), 65 classes.
  • DomainNet-126: 4 domains (Clipart, Painting, Real, Sketch), 126 classes.
  • DomainNet: Large-scale benchmark with 6 domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch), 345 classes.

Key Insights and Ideas

  • Less is More: Retaining only one or two highly transferable source domains frequently outperforms aggregating all sources. Pruning irrelevant sources prevents noise accumulation and negative transfer.
  • Source Redundancy is Real: In ablation experiments, discarding source samples whose distance to cluster centers exceeded the mean radius caused no drop in transfer accuracy, proving that massive datasets contain extensive redundant tail data.
  • Federated On-the-Fly Pruning Beats Two-Stage Distillation: Unlike distillation pipelines that train full models twice to rank domains, pruning federated source models during training cut compute time on Office-Home from ~47,143 seconds (for SSD) down to ~6,743 seconds, while reducing GPU memory from 12.4 GB to 8.2 GB.

Key Results

  • Office-Home: AutoS achieves 82.9% (and 83.2% in its source-free setup, AutoS/sf), outperforming multi-domain baselines like SSD (75.4%), Co-MDA (76.9%), and competitive foundation-model-assisted methods like FuzHDA (82.6%) and LCFD (83.2%).
  • DomainNet-126 & DomainNet: On DomainNet-126, AutoS reaches 83.2% average accuracy (surpassing SSD at 77.5% and FuzHDA at 82.0%). On the full DomainNet benchmark, AutoS using a frozen ViT backbone (AutoS*) achieves 47.9%, outperforming CDTrans (45.2%) and SSRT (45.2%).
  • Efficiency: Achieves a ~7× speedup in training time compared to standard distillation methods (SSD) while simultaneously cutting peak memory usage.

Limitations and Future Work

  • Closed-Set Assumption: The core method assumes identical category sets across all sources and targets. The authors identify extending AutoS to handle label shifts (such as open-set, partial, or universal domain adaptation) and modality shifts as key future directions.
  • Metric Sensitivity: Cluster thresholding is sensitive to the chosen distance metric. While average cosine radius performed best, root-mean-square and maximum radii over-included noisy target samples, leading to label degradation during model aggregation.

Summary updated

Abstract

Unsupervised multi-domain adaptation plays a key role in transfer learning by leveraging acquired rich source information from multiple source domains to solve target task from an unlabeled target domain. However, multiple source domains often contain much redundant or unrelated information which can harm transfer performance, especially when in massive-source domain settings. It is urgent to develop effective strategies for identifying and selecting the most transferable knowledge from massive source domains to address the target task. In this paper, we propose a multi-domain adaptation method named Autonomous Source Knowledge Selection (AutoS) to autonomosly select source training samples and models, enabling the prediction of target task using more relevant and transferable source information. The proposed method employs a density-driven selection strategy to choose source samples during training and to determine which source models should contribute to target prediction. Simulteneously, a pseudo-label enhancement module built on a pre-trained multimodal modal is employed to mitigate target label noise and improve self-supervision. Experiments on real-world datasets indicate the superiority of the proposed method.
BibTeX
@misc{li2025autonomous,
  author = {Keqiuyin Li and Jie Lu and Hua Zuo and Guangquan Zhang},
  title = {{Autonomous Source Knowledge Selection in Multi-Domain Adaptation}},
  year = {2025},
  eprint = {2512.14710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.14710},
}

Loading related content...

Figure