Abstract
Domain shift poses a significant challenge to the reliability of deep learning models in real-world deployments. Traditional Domain Generalization (DG) methods often struggle with "negative knowledge transfer" when attempting to learn a single domain-invariant representation. We present Meta-DMoE, a framework for unsupervised test-time adaptation (TTA) that treats specialized domain experts as teachers. By employing a transformer-based aggregator to meta-distill knowledge from a Mixture-of-Experts (MoE), Meta-DMoE enables a lightweight student model to rapidly adapt to unseen target domains using only a small set of unlabeled samples. Our approach achieves state-of-the-art performance on several real-world benchmarks, including the WILDS dataset, demonstrating superior adaptation speed and feature discriminability.
Introduction
Deep learning models typically operate under the assumption that training and testing data are independent and identically distributed (i.i.d.). However, in practical applications—such as medical imaging or wildlife monitoring—this assumption is frequently violated by domain shift. For instance, a model trained on North American wildlife may fail in tropical environments due to variations in lighting and background.
While Domain Generalization (DG) seeks to learn features that are invariant across domains, this often results in the loss of specialized, high-fidelity information. Test-time adaptation (TTA) addresses this by updating models on-the-fly, but many TTA methods are limited by their reliance on a single, potentially biased source model. Meta-DMoE overcomes these limitations by aggregating knowledge from multiple specialized experts.
Methodology
The Meta-DMoE framework consists of four primary components designed to facilitate efficient knowledge transfer and adaptation.
1. Mixture-of-Experts (MoE) as Teachers
Instead of a single global model, $N$ separate "expert" models are trained, each specialized on a specific source domain. This architecture preserves the unique nuances of each domain, preventing the "washing out" of specialized features.
2. Transformer-based Aggregator
At test time, a small batch of unlabeled data from the target domain is processed by all experts. A transformer encoder analyzes the relationships between the experts' features and the target data, dynamically learning to weight and combine their knowledge. This allows the system to "route" knowledge from the most relevant source domains to the target task.
3. Knowledge Distillation and Meta-Learning
The aggregated output serves as a supervision signal for a "student" network. The student is updated via distillation to match the teacher's aggregated features. To ensure this process generalizes to unseen domains, a bi-level meta-learning loop is used. During training, domain shifts are simulated by masking the expert corresponding to the current training domain, forcing the aggregator to learn how to generalize using other experts.
Experimental Results
Meta-DMoE was evaluated on challenging real-world benchmarks, including the WILDS collection (iWildCam, Camelyon17, RxRx1, FMoW, PovertyMap) as well as DomainNet and PACS.
Performance and Efficiency
Meta-DMoE outperformed state-of-the-art methods such as ARM, CORAL, and IRM on 4 out of 5 WILDS datasets. Notably, on the iWildCam dataset, it significantly improved Macro F1 scores. Unlike many TTA methods that require adaptation for every test sample, Meta-DMoE adapts once per domain, maintaining $O(1)$ inference complexity.
Feature Quality Analysis
Visualizing the feature space using t-SNE reveals the impact of the meta-distillation process. Compared to standard Empirical Risk Minimization (ERM), Meta-DMoE produces much tighter and more discriminative clusters for unseen target domains, indicating more robust feature representations.
Discussion and Conclusion
Meta-DMoE demonstrates that embracing domain specialization can be more effective than seeking domain invariance. By using a transformer to dynamically aggregate expert knowledge, the framework provides a flexible and privacy-preserving method for test-time adaptation. The use of distillation also allows for the deployment of lightweight student models on edge devices.
Future work will focus on improving scalability. Currently, the adaptation step requires a forward pass through all experts, which may become computationally expensive as the number of source domains increases. Exploring "sparse-gated" MoE architectures, where only the most relevant experts are activated, remains a promising direction for future research.
Summary updated