Keeping Replay Consistent for Continually Trained Parallel Translators

Non-autoregressive translation models write every output word at once instead of left to right, which makes them roughly fifteen times faster at inference. The price is that they cannot handle a sentence that has several valid translations: each position picks words independently, so the output can mix pieces of different correct answers. The standard fix is sequence-level knowledge distillation, where a slow left-to-right "teacher" model retranslates the training set so that every source sentence gets one consistent, simplified target. The parallel "student" then learns from those teacher outputs. Separately, continual learning, where a model is trained on one domain after another, relies on experience replay: a small memory of old examples is mixed into training on each new domain so the model does not forget the old ones.

This note argues that combining the two creates a failure neither literature has named. In a continual setting the teacher also changes from stage to stage. Replay targets distilled by last year's teacher and new-domain targets distilled by today's teacher then encode different preferred phrasings and lengths for similar inputs. To a parallel student, that mixture looks like the multi-reference data that distillation was supposed to remove. We propose to store only source sentences in the replay memory and to re-distill them with the current teacher at every stage, choosing per example between the fresh translation and a stored stale one. We also propose to test directly whether the benefit comes from cross-stage consistency rather than from target quality.

Core Claim

For a parallel student, the useful property of a replay buffer is not only that it covers old domains. The buffer and the incoming stream must also look as if they came from a single low-ambiguity teacher. On the Learning of Non-Autoregressive Transformers shows that a parallel model's error is bounded by the conditional total correlation of its training targets. This is the gap between the sum of per-word uncertainties and the uncertainty of the whole sentence, which is high exactly when a source has several distinct valid translations. The same paper shows that distillation works by building a proxy distribution with much lower total correlation, and that the proxy likelihood tracks BLEU with |r| ≈ 0.99. That analysis is entirely offline, with one teacher and one dataset.

Our claim is that, in continual training, stale distilled replay targets drive total correlation back up in a specific way. Across stages, similar sources end up paired with targets from two different teachers. The parallel student should suffer from this measurably more than a left-to-right student trained on the same data. Two things would be new. The first is naming and measuring this cross-stage inconsistency. The second is a replay design, source-only memory with consistency-aware re-distillation, that removes it without storing target text or token-level probability tables.

Research Program

The setting is a parallel student trained across a sequence of domains, together with a left-to-right teacher that is itself updated continually. The teacher is updated with ordinary replay and distillation in the style of Continual Learning for Neural Machine Translation. That paper handles left-to-right models with Dynamic Knowledge Distillation (an increasing weight on matching old-model predictions), a bias-correction module, and a 50,000-sample buffer. We reuse it as the teacher's training method, not as the contribution.

The memory stores a small reservoir-sampled set of source sentences per domain. Continual Learning with Tiny Episodic Memories supplies both the sampling scheme and the motivation for tiny buffers, since it shows that one to three stored examples per class beat regularization methods by 7–17% average accuracy. Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation shows that 50–100 unlabeled source sentences suffice for a left-to-right model. We keep the source-only idea but differ in two ways. We produce discrete target sequences rather than matching per-word soft probabilities, and our student is parallel. A parallel student needs a concrete target length to allocate output positions, which soft-label matching does not provide.

At each stage the current teacher beam-decodes the buffered sources. This introduces a tension. The current teacher gives consistent targets, but it may have partly forgotten the oldest domains, so its translations there can be worse. The translation stored when the example was first seen is better but inconsistent with the current stream. We resolve this per example with a selection rule adapted from One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation. That paper generates several teacher outputs and trains on whichever one the student currently finds easiest; we apply the same rule to choose between the stale and fresh candidates. The rule defaults to the fresh target unless the stale one's teacher-scored quality is higher by a margin. This requires storing one stale target per source, which is O(T) tokens, still far below the O(T·|V|) needed for soft labels.

The analysis half of the program measures the proposed mechanism directly. Exact total correlation is intractable. We therefore track three proxies. The first is the alignment-entropy complexity measure of Understanding Knowledge Distillation in Non-autoregressive Machine Translation, computed over the actual training mixture of buffer plus stream. The second is cross-stage agreement: BLEU between stale and fresh teacher outputs for the same source. The third is the spread of target-to-source length ratios. Following How Length Prediction Influence the Performance of Non-Autoregressive Translation?, which finds that distilled data concentrates length errors within ±5 tokens and that better length choice is worth about 3 BLEU, we also separate forgetting in the length predictor from forgetting in the word decoder. We do this by evaluating old domains with both predicted and oracle (teacher) lengths.

Baselines

The naive reference point is sequential fine-tuning of the parallel student on each new domain's distilled data with no memory. The upper reference is joint training of a parallel student on distilled data from all domains at once, which estimates the ceiling. No continual-learning result for parallel translators exists; I found no reported result on Lacuna for continual non-autoregressive translation on OPUS or any other stream. The closest reported numbers are for left-to-right models. Continual Learning for Neural Machine Translation reports that DKD+BiC with a 50,000-sample buffer beats fine-tuning by 3–4 BLEU on the general-domain test set. Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation reports an IT BLEU of 39.87 after the IT→Koran transition. We will reproduce both on our left-to-right teacher to calibrate the setup.

The strongest competitor, and the ablation that isolates our key ingredient, is stale distilled replay. Here each buffered source keeps the target produced by the teacher of its own stage, forever. This is what anyone would build first, and it already fixes the obvious problem of replaying raw human text. It will be run with the same buffer size, the same number of student updates, the same teacher checkpoints, and the same hyperparameter search budget as our method. Beating it means higher average BLEU on earlier domains at equal buffer size, with no loss on the current domain. To count as a real win, the improvement must exceed the seed-to-seed spread across three seeds, and it must coincide with lower measured complexity of the training mixture.

Two further baselines complete the picture. Raw-target replay stores human references, and fresh-only re-distillation drops the stale candidate. A student trained with glancing training, which feeds some correct words back to the decoder during training and so partly handles multiple references internally, will be run under every replay variant. This tests whether the consistency effect survives a stronger parallel architecture. Token-level soft-label replay in the style of SCD, adapted to the parallel student by conditioning on teacher lengths, is the alternative on the memory axis. We report it at its true memory cost.

Why not just…?

Why not just replay the distilled targets you stored at the time? This is the real threat. If the teacher barely drifts on sentences that resemble old domains, stale targets are already consistent and our method adds nothing. We do not assume the drift exists. Cross-stage agreement between stale and fresh targets is measured first. If it stays high, above the agreement between two beam-search runs of the same teacher, the proposal reduces to a negative result, which is still worth reporting. The decisive test appears in the experiments below.

Why not just replay human references, as ordinary experience replay does? Offline, Non-Autoregressive Neural Machine Translation and Understanding Knowledge Distillation in Non-autoregressive Machine Translation show that parallel models trained on raw text produce repeated and mixed-up words and lose substantial BLEU. We expect the same inside the replay stream, but we will include raw replay rather than assume it. A tiny raw buffer diluted in a large distilled stream might be harmless, and that would itself be informative.

Why not just keep one frozen general-domain teacher, so targets never drift? Consistency would then be perfect, but the teacher would translate medical or legal text poorly, and Understanding Knowledge Distillation in Non-autoregressive Machine Translation shows that student quality tracks teacher suitability closely. Its FlowSeq student moved from 18.55 to 22.0 BLEU just by changing the teacher. We run the frozen teacher as a baseline. If it matches our method, consistency would matter more than in-domain teacher quality, and that would be a simpler recommendation we would adopt.

Why not use parameter regularization such as EWC? On IWSLT17 European, Self-generated Replay Memories for Continual Neural Machine Translation reports 10.23 BLEU for EWC, against 8.75 for plain incremental training and 23.66 for its self-generated replay. That paper builds replay data by having a left-to-right model sample sentences and back-translate them. We include EWC only as a sanity point. We do not attempt self-generated replay from the parallel student itself, because such a student cannot sample coherent open-ended text. Our design sidesteps that problem by having the left-to-right teacher do all the generating.

Proposed Experiments

The main benchmark is the German–English OPUS multi-domain stream (IT, Koran, Law, Medical, Subtitles) as used by Continual Learning for Neural Machine Translation, starting from a model pretrained on WMT data. Buffer sizes range from 100 source sentences per domain up to the 50,000-sample setting of that paper. We use two domain orders, one of which switches abruptly between terse and verbose domains to stress the length predictor. Metrics are average BLEU over all seen domains after the final stage, the per-domain drop relative to the best score ever reached, current-domain BLEU, decoding speedup, and the length-oracle gap described above. A secondary run on the IWSLT17 European stream makes contact with the numbers reported by Self-generated Replay Memories for Continual Neural Machine Translation, although that stream is multilingual and less domain-shifted.

The decisive check is a controlled dose–response experiment. We hold teacher quality fixed by using only stale targets whose quality score is within a small band of the fresh ones. We then vary the fraction of buffer targets drawn from the stale teacher from 0 to 100%, and train both a parallel student and a left-to-right student of matched size on each mixture. The mechanism predicts two things. Parallel-student forgetting should rise with the stale fraction, while the left-to-right student should stay roughly flat. The rise should also track the measured complexity of the training mixture. If both student types degrade equally, the effect is ordinary target-quality noise, not multimodality, and the core claim is wrong. A second check separates the two components. If oracle lengths recover most of the old-domain loss under stale replay, forgetting lives mainly in the length predictor. Consistent re-distillation should then shrink the oracle gap specifically, which would locate the benefit mechanistically rather than only in the final BLEU.