When a multilingual model is fine-tuned on one language after another, it forgets the earlier ones. The standard remedy is to keep a small store of old examples and mix them back in, which is called replay. Storing user utterances is often not allowed, so a privacy-friendly alternative is generative replay. Here a generator learns to produce imitation examples of earlier languages, and those are replayed instead of real data. Generative replay has a known weakness: the imitations are imperfect. In text, imitations of low-resource languages are especially likely to be garbled, mislabeled, or off-distribution. Training on bad imitations can hurt more than it helps, and adding more of them can make things worse.
We propose to use the model's own uncertainty to decide which generated examples are worth replaying. Uncertainty comes in two kinds. Epistemic uncertainty comes from missing knowledge and can be fixed by more training. Aleatoric uncertainty comes from noise or ambiguity in the input itself and cannot be fixed by training. Our claim is that the two kinds separate the two things that matter in generated replay. A generated sentence in an old language deserves replay if the current model has become epistemically unsure about it, which signals forgetting. It deserves rejection if a snapshot of the model saved right after learning that language finds it noisy or confusing, which signals a generator artifact. Gating replay by this decomposition should make generative replay competitive in cross-lingual continual learning. It should also explain the published finding that replaying "easy" examples beats replaying "hard" ones.
Core Claim
Prior work in this area points in two directions that seem to conflict. Leitner-Guided Memory Replay for Cross-lingual Continual Learning keeps a buffer of real examples and gives each one a 1–5 mastery score. It finds that replaying well-mastered examples beats replaying poorly mastered ones, cutting MTOP slot-filling forgetting by 1.76 points. The authors attribute this to "hard" cross-lingual examples often being noise. By contrast, Prioritized Generative Replay uses a diffusion generator in reinforcement learning. Steering it toward the agent's learning frontier, the visited-but-not-mastered states, roughly doubles return on a sparse control task. We read both results as instances of one rule: replay what is learnable and currently being lost, and drop what is noise.
The mastery score in the Leitner paper cannot tell those two cases apart. The epistemic/aleatoric split can. In generative replay, the dominant noise source is the generator itself, so the aleatoric side of the gate doubles as a fidelity filter.
The concrete gate scores a generated example $x$ in an old language with predicted label $y$. It uses two signals. The first, $U_{\text{epi}}^{\text{cur}}(x)$, is the current model's epistemic uncertainty, estimated as the disagreement across Monte Carlo dropout passes, i.e. the mutual information between prediction and weights. The second, $U^{\text{snap}}(x)$, is the total predictive uncertainty of the frozen snapshot taken when that language was last trained. We keep $x$ when
$$U^{\text{snap}}(x) < \tau_{\text{fid}} \quad \text{and} \quad U_{\text{epi}}^{\text{cur}}(x) > \tau_{\text{forget}},$$
where $\tau_{\text{fid}}$ and $\tau_{\text{forget}}$ are thresholds set by quantile on held-out generated samples. The first condition says the model that truly knew the language finds the sample clean. The second says the current model has drifted on it.
This pairing borrows its logic from ACED (Udandarao et al.), which selects samples where a student is unsure and a teacher is sure. Here the "teacher" is the solver's own past self, and it doubles as the source of pseudo-labels.
Research Program
We build on Prototype Conditioned Generative Replay for Continual Learning in NLP, which supplies the generator. That method conditions a variational autoencoder on a Gaussian "prototype": the mean and spread of a task's hidden representations. It uses a drift-correction step, Prototype Shift Estimation, to keep old prototypes aligned as the backbone changes. It reached an average performance of 59.48 on a 12-task English sequence, but it was never tested across languages and samples latents uniformly from the prototype.
We make three changes. First, we keep one prototype per language and task. Second, we label each generated utterance with the frozen snapshot rather than trusting the conditioning label. Third, we apply the gate above, then resample latents near accepted samples so generation concentrates where forgetting is happening. The solver is a multilingual encoder of the XLM-R kind used in MTOP. The language stream and evaluation follow Cross-lingual Continual Learning, which measures four things: retention of earlier languages, speed of learning later ones, zero-shot transfer to held-out languages, and final overall accuracy.
The work is also an analysis study. Rong et al. (2025) report that with low-fidelity replay, accuracy first drops as replay volume grows. Cross-lingual Continual Learning finds that low-to-high-resource orderings forget far more than high-to-low. We expect generator fidelity to be worse for low-resource and script-distant languages such as Thai and Hindi in MTOP. Two predictions follow. The ungated volume dip should be deepest for those languages, and the gate should reject a larger share of their samples. If either prediction fails, the fidelity story behind the gate is wrong, even if the method happens to work.
Baselines
The naive reference is sequential fine-tuning with no protection. Cross-lingual Continual Learning reports a forgetting score of 2.93 for it on MTOP. The forgetting floor is language-specific Transformers, reported at 0.49 forgetting in the same paper with poor zero-shot generalization. We include it to show the trade-off rather than as a target. The ceiling is joint multilingual training on all languages at once. For an MTOP continual-learning number for joint training, no reported result was found on Lacuna. For a per-language accuracy reference, MTOP reports 77.7% exact match for in-language XLM-R.
The strongest existing competitor is Leitner-guided experience replay with the "easy" policy from Leitner-Guided Memory Replay for Cross-lingual Continual Learning. It is the best published replay method on this exact benchmark and stream. The file gives only its 1.76-point margin over the "hard" policy, not an absolute score, so we rerun it ourselves. We test it at several buffer sizes, which yields a curve of forgetting versus stored real examples per language.
The strongest generative competitor is the original prototype-conditioned replay, transplanted to the multilingual stream with uniform latent sampling and no gate. This isolates what the gate adds.
Fairness is enforced through budgets and settings. All replay methods see the same number of replayed examples per training step, run for the same number of steps, and get the same hyperparameter search budget of 20 trials on a development stream. The generative methods additionally share the same generator architecture and the same generator training compute. We also tally the extra forward passes the gate needs for dropout sampling and report wall-clock time.
Three ablations isolate the key ingredient. The first keeps only the fidelity condition. The second keeps only the forgetting condition. The third replaces the decomposed gate with a single confidence threshold, which amounts to "easy-only" filtering of generated text.
Beating ungated prototype replay on preservation and final utility in both orders, without losing zero-shot generalization, counts as a win for the gate. Matching Leitner-easy replay at a given buffer size counts as a win for the privacy setting, and the buffer size at which it matches is the headline number.
Why not just…?
Why not just keep a small buffer of real examples? This is the most serious threat. A few hundred real utterances per language may simply beat any generator, and nothing in the file shows generative replay winning on text at this scale. Indeed, van de Ven et al. (2022) find generative replay degrades as data gets more complex. We do not argue around this. We report the exchange rate: how many real exemplars per language gated generative replay is worth. If the answer is "fewer than ten," the method is useful only where storing text is forbidden, and we will say so. We also test a hybrid with a tiny real buffer. Prioritized Generative Replay found performance collapses when synthetic data exceeds about 87% of replay, so some real data may be essential.
Why not just filter generated samples by confidence, the way Leitner-easy replay does? This is the simplest rival and is exactly our third ablation. Confidence filtering removes garbage, but it also removes the samples the current model has started to forget, since those are by definition low-confidence for the current model. Our gate separates "the snapshot finds it noisy" from "the current model has drifted on it." If confidence filtering matches the decomposed gate, the decomposition is unnecessary, and the contribution shrinks to an empirical finding about filtered generative replay.
Why not just translate current-language training data into the old languages and replay that? Translation replay needs no generator and produces fluent text. However, MTOP documents slot-boundary errors from machine translation, especially in unsegmented Thai. It also needs a translation system for every old language. We run translate-replay as a baseline under the same replay budget, labeling translated utterances with the snapshot as well. If it wins, the gate can be applied to translated samples instead of generated ones. That is a cheaper variant we would then recommend.
Proposed Experiments
The main experiment runs the six-language MTOP stream in both the high-to-low and low-to-high orders, over three seeds and three random language permutations. It reports the four multi-hop measures with intent accuracy and slot F1. MultiATIS++, used by the Leitner paper, serves as a second stream to check that conclusions are not MTOP-specific. Zero-shot generalization is measured on languages held out of the stream, following the multi-hop protocol.
Three decisive checks follow. First, a volume sweep varies replayed examples per step over an order of magnitude. Ungated generation should reproduce the non-monotonic dip of Rong et al. (2025), and gated generation should not. Second, we audit the gate's rejections. On 300 accepted and 300 rejected generated utterances in Hindi, Thai, and Spanish, bilingual annotators judge fluency and label correctness. The fidelity condition is supported only if rejected samples are clearly worse. Third, we repeat the easy-versus-hard comparison on generated data, once with mastery scores and once with the decomposed split. If the Leitner result is really about noise, "hard" should lose under mastery scores but "high-epistemic, low-snapshot-uncertainty" should win. If that inversion does not appear, the unifying claim is false, and the paper reports the negative result.
References
- Leitner-Guided Memory Replay for Cross-lingual Continual Learning
- Prioritized Generative Replay
- Active Data Curation Effectively Distills Large-Scale Multimodal Models
- Prototype Conditioned Generative Replay for Continual Learning in NLP
- MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
- Cross-lingual Continual Learning
- CAN: Leveraging Clients As Navigators for Generative Replay in Federated Continual Learning
- Three types of incremental learning