Expert Routing with Synthetic Data for Domain Incremental Learning

Abstract

In many real-world settings, regulations and economic incentives permit the sharing of models but not data across institutional boundaries. In such scenarios, practitioners might hope to adapt models to new domains, without losing performance on previous domains (so-called catastrophic forgetting). While any single model may struggle to achieve this goal, learning an ensemble of domain-specific experts offers the potential to adapt more closely to each individual institution. However, a core challenge in this context is determining which expert to deploy at test time. In this paper, we propose Generate to Discriminate (G2D), a domain-incremental learning method that leverages synthetic data to train a domain-discriminator that routes samples at inference time to the appropriate expert. Surprisingly, we find that leveraging synthetic data in this capacity is more effective than using the samples to directly train the downstream classifier. We observe that G2D outperforms competitive domain-incremental learning methods on tasks in both vision and language modalities, providing a new perspective on the use of synthetic data in the lifelong learning literature.

Introduction

In ideal machine learning scenarios, access to a comprehensive, centralized dataset allows for the training of robust models. However, real-world constraints—particularly in sensitive sectors like healthcare—often silo data within institutional boundaries. A model may sequentially learn from different hospitals, but legal and ethical barriers frequently prevent it from revisiting data from previous domains.

This environment presents two primary challenges in Domain Incremental Learning (DIL):

1. Catastrophic Forgetting: Sequential learning often causes a model to overwrite weights optimized for previous domains.
2. The Routing Dilemma: While an "ensemble of experts" (one model per domain) can mitigate forgetting, the system must identify the correct expert at inference time without being explicitly told the domain of the input sample.

Traditional approaches often rely on Generative Replay, using synthetic data to retrain classifiers. However, the inherent noise in synthetic samples can degrade classification performance.

The Generate to Discriminate (G2D) Framework

The proposed Generate to Discriminate (G2D) framework shifts the role of synthetic data. Instead of using it to help a model classify labels, G2D uses it to help the model recognize the domain of an input.

Expand

Source, p. 2

For each new domain $t$, the workflow is as follows:

1. Generator Fine-tuning: A generative model (e.g., Stable Diffusion or T5) is trained on the current domain's data.
2. Synthetic Data Generation: The generator produces samples representing the domain's distribution.
3. Expert Training: A domain-specific classifier (expert) is fine-tuned on real data. To maintain efficiency, Low-Rank Adaptation (LoRA) is employed, updating only a small fraction of parameters.
4. Discriminator Training: A domain discriminator is trained on the union of all synthetic data generated across domains. It learns to distinguish between domain-specific "styles."
5. Inference Routing: At test time, the discriminator predicts the domain ID of an incoming sample, routing it to the corresponding expert for final prediction.

Experimental Evaluation

G2D was evaluated across diverse benchmarks, including standard vision tasks (DomainNet, CORe50) and language tasks (SQuAD, TriviaQA, QuAC). Additionally, the authors introduced DermCL, a dermatology benchmark comprising four clinical datasets (HAM10000, BCN2000, PAD-UEFS-20, and DDI) with realistic distribution shifts and diverse skin tones.

Key Insights

A central finding is that discriminators are more robust than classifiers when trained on synthetic data. While synthetic samples may contain artifacts that mislead a classifier, they still capture the general "style" or "distribution" of a domain. Consequently, it is easier for a model to learn domain boundaries than to learn fine-grained label boundaries from synthetic data.

Expand

Source, p. 8

Results

G2D consistently outperformed baselines:

• DomainNet: G2D achieved 70.03% average accuracy, significantly higher than Generative Replay (60.16%) and S-Prompts (56.13%).
• DermCL: G2D reached an 85.24% AUC, outperforming the next best method (81.59%).
• Text QA: G2D achieved a 64.7 F1 score, nearing the multi-task upper bound of 68.6.
• Routing Accuracy: The discriminator achieved 94-97% accuracy, nearly matching an "Oracle" trained on real data.

Conclusion and Future Work

G2D provides a novel perspective on synthetic data in lifelong learning. While effective, the current approach scales linearly with the number of domains, which may pose storage challenges at extreme scales. Future research could explore expert merging to consolidate similar models. Additionally, integrating Differential Privacy during generative training could further address potential data leakage concerns in sensitive applications.