Abstract
In many real-world settings, regulations and economic incentives permit the sharing of models but not data across institutional boundaries. In such scenarios, practitioners might hope to adapt models to new domains, without losing performance on previous domains (so-called catastrophic forgetting). While any single model may struggle to achieve this goal, learning an ensemble of domain-specific experts offers the potential to adapt more closely to each individual institution. However, a core challenge in this context is determining which expert to deploy at test time. In this paper, we propose Generate to Discriminate (G2D), a domain-incremental continual learning method that leverages synthetic data to train a domain-discriminator that routes samples at inference time to the appropriate expert. Surprisingly, we find that leveraging synthetic data in this capacity is more effective than using the samples to directly train the downstream classifier (the more common approach to leveraging synthetic data in the lifelong learning literature). We observe that G2D outperforms competitive domain-incremental learning methods on tasks in both vision and language modalities, providing a new perspective on the use of synthetic data in the lifelong learning literature.
Introduction
Machine learning models deployed in the real world often encounter data from a sequence of distinct environments or domains. For instance, a medical diagnostic tool might be sequentially introduced to different hospitals, each with unique patient demographics and imaging equipment. A primary challenge in this setting is continual learning: adapting to new domains without suffering from "catastrophic forgetting" of previous ones.
Traditional approaches are often hindered by two factors. First, privacy regulations frequently prohibit the sharing of raw data between institutions, making it impossible to retrain a centralized global model. Second, while training domain-specific "experts" can preserve performance, deploying them effectively requires knowing which expert to use for a given test sample—a "routing problem" that is difficult when domain identifiers are missing at inference time.
The G2D Framework
To address these challenges, we introduce Generate to Discriminate (G2D). Rather than attempting to build a single generalist model, G2D maintains an ensemble of domain-specific experts and utilizes synthetic data to solve the routing problem.
As illustrated in the framework overview, the process involves three main components:
- Domain Experts: For each new domain, a specific expert classifier is fine-tuned.
- Generative Modeling: Simultaneously, a generative model (such as Stable Diffusion for images or T5 for text) is trained on the current domain's data.
- The Router: The generative models for all encountered domains are used to produce synthetic data. A domain discriminator (the router) is then trained on this synthetic data to identify the domain of any given input.
At inference time, an unlabeled sample is first passed to the router, which identifies the most likely domain. The sample is then routed to the corresponding expert for final classification.
Experimental Evaluation
We evaluated G2D across diverse benchmarks in both vision and language modalities:
- Vision: DomainNet (diverse styles like sketches and paintings) and CORe50 (object recognition).
- Language: A Question Answering (QA) benchmark including SQuAD, TriviaQA, and QuAC.
- DermCL: A new, realistic dermatology benchmark consisting of four clinical datasets (HAM10000, BCN2000, PAD-UEFS-20, and DDI) with significant distribution shifts.
Key Results and Insights
G2D consistently outperformed competitive baselines, including generative replay and state-of-the-art prompt-based methods. On DomainNet, G2D achieved 70.03% average accuracy, significantly higher than the 56.13% achieved by S-Prompts. In the medical DermCL benchmark, G2D reached an 85.24% AUC, outperforming S-Prompts (81.59%) and simple sequential fine-tuning (70.49%).
A critical finding of this work is that discrimination is easier than rehearsal. While generative models may struggle to capture the fine-grained details necessary for high-accuracy classification (generative replay), they are remarkably effective at capturing the broader "style" or distribution of a domain. This allows the G2D router to achieve high accuracy—reaching 97.38% on CORe50—enabling near-optimal expert selection.
Conclusion
G2D demonstrates that synthetic data is most effectively used in continual learning as a tool for domain discrimination rather than direct classifier training. By decoupling domain identification from task classification, G2D provides a modular, high-performance solution for domain-incremental learning that respects data privacy constraints. Future work will focus on improving the efficiency of the expert ensemble and integrating differential privacy to further secure the generative process.