Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts

Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, Jin Tang

Abstract

Domain shift poses a significant challenge to the reliability of deep learning models in real-world deployments. Traditional Domain Generalization (DG) methods often struggle with "negative knowledge transfer" when attempting to learn a single domain-invariant representation. We present Meta-DMoE, a framework for unsupervised test-time adaptation (TTA) that treats specialized domain experts as teachers. By employing a transformer-based aggregator to meta-distill knowledge from a Mixture-of-Experts (MoE), Meta-DMoE enables a lightweight student model to rapidly adapt to unseen target domains using only a small set of unlabeled samples. Our approach achieves state-of-the-art performance on several real-world benchmarks, including the WILDS dataset, demonstrating superior adaptation speed and feature discriminability.

Introduction

Deep learning models typically operate under the assumption that training and testing data are independent and identically distributed (i.i.d.). However, in practical applications—such as medical imaging or wildlife monitoring—this assumption is frequently violated by domain shift. For instance, a model trained on North American wildlife may fail in tropical environments due to variations in lighting and background.

While Domain Generalization (DG) seeks to learn features that are invariant across domains, this often results in the loss of specialized, high-fidelity information. Test-time adaptation (TTA) addresses this by updating models on-the-fly, but many TTA methods are limited by their reliance on a single, potentially biased source model. Meta-DMoE overcomes these limitations by aggregating knowledge from multiple specialized experts.

Methodology

The Meta-DMoE framework consists of four primary components designed to facilitate efficient knowledge transfer and adaptation.

1. Mixture-of-Experts (MoE) as Teachers

Instead of a single global model, $N$ separate "expert" models are trained, each specialized on a specific source domain. This architecture preserves the unique nuances of each domain, preventing the "washing out" of specialized features.

2. Transformer-based Aggregator

At test time, a small batch of unlabeled data from the target domain is processed by all experts. A transformer encoder analyzes the relationships between the experts' features and the target data, dynamically learning to weight and combine their knowledge. This allows the system to "route" knowledge from the most relevant source domains to the target task.

3. Knowledge Distillation and Meta-Learning

The aggregated output serves as a supervision signal for a "student" network. The student is updated via distillation to match the teacher's aggregated features. To ensure this process generalizes to unseen domains, a bi-level meta-learning loop is used. During training, domain shifts are simulated by masking the expert corresponding to the current training domain, forcing the aggregator to learn how to generalize using other experts.

Overview of the Meta-DMoE training process. Knowledge is queried from experts, combined by an aggregator, and distilled into a student model.

Experimental Results

Meta-DMoE was evaluated on challenging real-world benchmarks, including the WILDS collection (iWildCam, Camelyon17, RxRx1, FMoW, PovertyMap) as well as DomainNet and PACS.

Performance and Efficiency

Meta-DMoE outperformed state-of-the-art methods such as ARM, CORAL, and IRM on 4 out of 5 WILDS datasets. Notably, on the iWildCam dataset, it significantly improved Macro F1 scores. Unlike many TTA methods that require adaptation for every test sample, Meta-DMoE adapts once per domain, maintaining $O(1)$ inference complexity.

Feature Quality Analysis

Visualizing the feature space using t-SNE reveals the impact of the meta-distillation process. Compared to standard Empirical Risk Minimization (ERM), Meta-DMoE produces much tighter and more discriminative clusters for unseen target domains, indicating more robust feature representations.

t-SNE visualization comparing ERM (baseline) to Meta-DMoE across Camelyon17 and iWildCam datasets. Meta-DMoE (Ours) shows clearer class separation.

Discussion and Conclusion

Meta-DMoE demonstrates that embracing domain specialization can be more effective than seeking domain invariance. By using a transformer to dynamically aggregate expert knowledge, the framework provides a flexible and privacy-preserving method for test-time adaptation. The use of distillation also allows for the deployment of lightweight student models on edge devices.

Future work will focus on improving scalability. Currently, the adaptation step requires a forward pass through all experts, which may become computationally expensive as the number of source domains increases. Exploring "sparse-gated" MoE architectures, where only the most relevant experts are activated, remains a promising direction for future research.

Summary updated

Abstract

In this paper, we tackle the problem of domain shift. Most existing methods perform training on multiple source domains using a single model, and the same trained model is used on all unseen target domains. Such solutions are sub-optimal as each target domain exhibits its own specialty, which is not adapted. Furthermore, expecting single-model training to learn extensive knowledge from multiple source domains is counterintuitive. The model is more biased toward learning only domain-invariant features and may result in negative knowledge transfer. In this work, we propose a novel framework for unsupervised test-time adaptation, which is formulated as a knowledge distillation process to address domain shift. Specifically, we incorporate Mixture-of-Experts (MoE) as teachers, where each expert is separately trained on different source domains to maximize their specialty. Given a test-time target domain, a small set of unlabeled data is sampled to query the knowledge from MoE. As the source domains are correlated to the target domains, a transformer-based aggregator then combines the domain knowledge by examining the interconnection among them. The output is treated as a supervision signal to adapt a student prediction network toward the target domain. We further employ meta-learning to enforce the aggregator to distill positive knowledge and the student network to achieve fast adaptation. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art and validates the effectiveness of each proposed component. Our code is available at https://github.com/n3il666/Meta-DMoE.
BibTeX
@inproceedings{zhong2022meta,
  author = {Tao Zhong and Zhixiang Chi and Li Gu and Yang Wang and Yuanhao Yu and Jin Tang},
  title = {{Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure