TL;DR
When adapting models across multiple source domains, more data isn't always better: irrelevant or noisy sources often degrade transfer performance. AutoS (Autonomous Source Knowledge Selection) solves this by dynamically pruning uninformative source domains and samples using a density-driven selection metric in a federated training setup, followed by prompt-tuned self-supervision from a frozen vision-language model (CLIP). This allows models to achieve state-of-the-art multi-domain adaptation performance using only the most transferable source knowledge at a fraction of the computational and memory cost.
What Problem Does This Paper Address?
Unsupervised multi-domain adaptation (MDA) aims to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Traditional methods combine all available sources—often via simple feature alignment or linear parameter averaging.
However, in massive multi-source settings, many domains or sample clusters contain redundant or completely irrelevant information that triggers negative transfer. Prior attempts to filter source domains (such as sample and source distillation) relied on manual heuristics or required training source models twice—once to evaluate domain similarity and once for actual adaptation—making them computationally prohibitive. AutoS tackles the challenge of autonomously identifying and pruning unhelpful source knowledge on the fly during training.
Methods
AutoS operates in a two-stage pipeline:
[Massive Source Domains + Target]
│
▼ (Stage 1: Federated Source Training & Autonomous Pruning)
• Single shared backbone across sources
• Compute source cluster centers and radii (Rd)
• Filter source/target confident samples based on distance thresholds
• Compute target data density (ρ) and source importance weights (ω)
• Prune irrelevant sources (Keep(D_s) thresholding)
│
▼ (Stage 2: Target Model Adaptation)
• Weighted aggregation of surviving source models
• Cross-modal guidance from frozen CLIP (prompt tuning via information bottleneck loss Lex)
• Target self-supervision (Lin)
- Density-Controlled Autonomous Selection (Stage 1): Instead of training isolated source models, AutoS initializes a single shared model updated in a federated manner. It computes class cluster centers from classifier weights and calculates cluster radii. It forms an intermediate domain by retaining confident source samples and pulling in high-confidence target samples. Source domains are evaluated via a composite weight combining target sample assignment ratio and cluster density; domains falling below a dynamic threshold are pruned mid-training.
- Cross-Modality Target Adaptation (Stage 2): Surviving source models are aggregated into a target model using their domain weights. To bridge remaining domain gaps without requiring raw source data matching, AutoS uses a frozen foundation model (CLIP). It optimizes learnable text prompts through a Gaussian structural causal/information-bottleneck loss ($L_{ex}$) and adapts the target model via entropy and self-supervision objectives ($L_{in}$).
Datasets
AutoS was evaluated on four standard benchmark datasets (no new benchmarks were introduced):
- Office-31: 3 domains (Amazon, DSLR, Webcam), 31 classes.
- Office-Home: 4 domains (Art, Clipart, Product, Real-World), 65 classes.
- DomainNet-126: 4 domains (Clipart, Painting, Real, Sketch), 126 classes.
- DomainNet: Large-scale benchmark with 6 domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch), 345 classes.
Key Insights and Ideas
- Less is More: Retaining only one or two highly transferable source domains frequently outperforms aggregating all sources. Pruning irrelevant sources prevents noise accumulation and negative transfer.
- Source Redundancy is Real: In ablation experiments, discarding source samples whose distance to cluster centers exceeded the mean radius caused no drop in transfer accuracy, proving that massive datasets contain extensive redundant tail data.
- Federated On-the-Fly Pruning Beats Two-Stage Distillation: Unlike distillation pipelines that train full models twice to rank domains, pruning federated source models during training cut compute time on Office-Home from ~47,143 seconds (for SSD) down to ~6,743 seconds, while reducing GPU memory from 12.4 GB to 8.2 GB.
Key Results
- Office-Home: AutoS achieves 82.9% (and 83.2% in its source-free setup,
AutoS/sf), outperforming multi-domain baselines like SSD (75.4%), Co-MDA (76.9%), and competitive foundation-model-assisted methods like FuzHDA (82.6%) and LCFD (83.2%). - DomainNet-126 & DomainNet: On DomainNet-126, AutoS reaches 83.2% average accuracy (surpassing SSD at 77.5% and FuzHDA at 82.0%). On the full DomainNet benchmark, AutoS using a frozen ViT backbone (
AutoS*) achieves 47.9%, outperforming CDTrans (45.2%) and SSRT (45.2%). - Efficiency: Achieves a ~7× speedup in training time compared to standard distillation methods (SSD) while simultaneously cutting peak memory usage.
Limitations and Future Work
- Closed-Set Assumption: The core method assumes identical category sets across all sources and targets. The authors identify extending AutoS to handle label shifts (such as open-set, partial, or universal domain adaptation) and modality shifts as key future directions.
- Metric Sensitivity: Cluster thresholding is sensitive to the chosen distance metric. While average cosine radius performed best, root-mean-square and maximum radii over-included noisy target samples, leading to label degradation during model aggregation.
Summary updated