Replaying Against the Forgetting Directions in Domain Continual Pre-Training

When a general language model is adapted to medicine by continuing its pre-training on medical text, it first gets worse at general reasoning and then partly recovers. This is called the stability gap. The standard fix is to mix a fixed fraction of random general web text back into training, which is called replay. Replay works, but it is blunt: every replayed document is chosen without regard to what the model is actually losing. This proposal asks whether the general capability lost during domain continual pre-training is concentrated in a few directions. If it is, the proposal asks whether replay documents can be chosen to push back along exactly those directions, using training data attribution: methods that estimate how much a given training document changes the model's loss on a given test example.

The concrete method has three steps. Track how the model's loss changes on a fixed set of general probe texts as medical training proceeds. Find the few dominant patterns in those changes, which we call forgetting modes. Then pick replay documents whose gradients most reduce loss along those modes. If forgetting is low-rank, a small, targeted replay budget should give the general-capability retention of a much larger uniform one, leaving more of the compute for medical tokens. If forgetting is not low-rank, we will have measured that for decoder language models for the first time, and the method falls back to a simpler gradient-alignment rule that we test as an ablation.

Core Claim

Catastrophic Forgetting is Low-Rank proves, using the neural tangent kernel (a linearization that describes how a network's outputs move under a gradient step), that forgetting concentrates in 1 to 6 function-space modes carrying 50–90% of forgetting energy. Its predictor reaches cosine similarity above 0.99 with actual forgetting. Its proof and tests, however, cover frozen vision backbones with linear heads under squared-error loss. That paper motivates this proposal and also bounds it: it says nothing about cross-entropy next-token prediction with all weights trained. The empirical domain continual pre-training literature reports broad drops across general tasks during the gap (Efficient Domain Continual pretraining by Mitigating the Stability Gap), which could mean either many independent losses or a few shared directions showing up across many tasks.

Our claim has two parts. First, measured on teacher-forced per-sequence losses over a general probe set, forgetting during medical continual pre-training of 1B–3B decoders is dominated by a handful of modes. Second, attribution scores computed against those modes select replay data that closes the stability gap with substantially fewer replay tokens than uniform replay or than attribution against average probe loss.

We deliberately leave the domain side of training unchanged. That isolates the contribution to replay selection, the part that current recipes handle with no model awareness at all.

Relation to Prior Work

Model-aware data selection already exists for training from scratch. MATES trains a small surrogate to predict the one-step loss reduction a candidate document produces on a single reference task (LAMBADA). Group-MATES extends this to interactions between documents on DCLM. Both select data to improve a model being built, from scratch, against one aggregate target. Neither selects replay data to protect an existing model, and neither breaks the target into separate failure directions. We reuse their machinery: one-step loss-reduction oracles and an optional distilled scorer.

The closest domain work is Guo et al., who mitigate the stability gap with KenLM n-gram perplexity filtering, multi-epoch training on a curated subset, and a fixed ratio of C4 and Common Crawl replay. They report OpenLlama-3B reaching 40.7% average medical accuracy with 40% of the training budget, and Llama-3-Physician-8B-Instruct reaching 76.7% across PubMedQA, MedMCQA, MedQA-4-Option, and MMLU-Medical. Their replay choice is model-blind. That gap is the one we target.

For computing gradient similarities cheaply we borrow from attribution work. Random projections of gradients, as in Clustered Influence Functions, and first-order trajectory scores, as in TracIn, both apply. We avoid inverse-Hessian influence (Understanding Black-box Predictions via Influence Functions) because a model mid-way through continual pre-training is far from a minimum, the regime in which that approximation degrades.

Two results constrain the design. Distributional Training Data Attribution shows that single-run attribution is noisy under seed randomness, so we estimate modes across checkpoints and seeds rather than from one gradient snapshot. Barriers to Counterfactual Credit Attribution for Autoregressive Models argues that credit for sampled generations is ill-posed, so we attribute only teacher-forced losses on fixed probe text, never generated outputs.

Research Program

The setup follows Guo et al.: continual pre-training of a 1B model and OpenLlama-3B on PubMed abstracts, the corpus BioBART used (BioBART), plus their KenLM-filtered medical web subset. Replay candidates come from FineWeb (The FineWeb Datasets) and C4.

The general probe set $P$ holds $m$ sequences, a few thousand, drawn from the training portions of the general suite used by MATES (SciQ, ARC, HellaSwag, PIQA and others) plus held-out FineWeb text. A disjoint set of general benchmarks is reserved for evaluation only.

The first stage is diagnostic. At each checkpoint $t$, we record the forgetting vector $\Delta_t = \ell_t(P) - \ell_0(P) \in \mathbb{R}^m$, the per-probe increase in loss relative to the base model. We stack these across checkpoints and three seeds into a matrix $F$. Its singular value spectrum tells us how many modes $k$ capture most of the forgetting energy.

In parallel, we test the first-order kernel predictor in the language-model setting. We compare the predicted one-step change $-\eta\, J_P\, g_D$ against the observed change, using projected gradients. Here $J_P$ is the matrix of projected per-probe gradients, $g_D$ is the projected gradient of the current domain batch, and $\eta$ is the learning rate. This tells us whether the vision paper's predictor survives cross-entropy training with all weights updated.

The second stage is selection. Let $v_1,\dots,v_k$ be the top left singular vectors of $F$, re-estimated every few thousand steps. These are directions in probe space. For a replay candidate $r$ with projected gradient $g_r$, the score is

$$s(r) = \sum_{j=1}^{k} \sigma_j \, v_j^\top J_P\, g_r ,$$

where $\sigma_j$ is the $j$-th singular value. The score measures how much one step on $r$ would reduce probe loss along the dominant forgetting modes, weighted by how much forgetting each mode carries. Replay slots in each batch are filled by Gumbel-top-$k$ sampling over $s$, following MATES, to keep the replay diverse.

Scoring every candidate requires one backward pass per candidate. If that proves too expensive, we distill the scores into a small regressor in the manner of MATES's surrogate, whose overhead MATES reports as 11.5% of FLOPs at 1B. Selection overhead is charged against the method in all comparisons.

Baselines

The naive reference is domain continual pre-training with no replay. It establishes the depth and duration of the stability gap and the medical ceiling when all compute goes to domain tokens.

The strongest existing recipe is Guo et al.'s: KenLM-filtered domain data with uniform C4 and Common Crawl replay at their fixed ratio. We run it with the same domain data, total tokens, learning-rate schedule, and three seeds. Their reported figure for this setting is 40.7% average medical accuracy for OpenLlama-3B at 40% budget. The file gives no isolated general-retention number for their replay, so that number will be measured. To beat this baseline, our method must match its general-benchmark retention and stability-gap area while using at most a third of its replay tokens, or improve medical accuracy at equal retention.

The key ablation is mean-probe gradient alignment. It is identical to our method except that $s(r)$ is the projected gradient's alignment with the average probe loss, as in a MATES-style oracle applied to replay. It has the same data, compute, and refresh schedule. If the low-rank decomposition matters, our method should beat this ablation at small replay budgets. If they tie, the low-rank framing is unnecessary for selection.

A non-gradient competitor picks replay documents by embedding similarity to the probe set. It tests whether gradients add anything beyond semantic closeness.

A parameter-penalty competitor adds an EWC-style Fisher penalty with uniform replay. It is tuned with the same hyperparameter search budget, eight configurations per method.

Why not just…?

Why not just replay more uniform general text? It may work, and it sets the bar. The cost is that every replay token displaces a domain token, and generic text can dilute domain patterns, the drowning effect reported in A Deep Fusion Model for Domain Adaptation in Phrase-based MT and MonoCoder. We sweep the uniform replay ratio to trace a medical-versus-general trade-off curve and ask whether targeted replay sits strictly above it. If it does not, the answer is to replay more, and we will report that.

Why not just align replay gradients with the average general-probe loss, without modes? This is the real threat. If forgetting in language models is broad rather than low-rank, the two rules coincide, and the simpler one wins on simplicity. The stage-one spectrum predicts the outcome before the selection runs. A flat spectrum predicts a tie. A sharp spectrum predicts our advantage at small budgets. The mean-alignment ablation settles the question either way.

Why not just regularize the weights with EWC instead of choosing data? The low-rank forgetting paper found that parameter-space penalties reached only about 20% on shared-head Split-MNIST, against about 78% for function-space regularization. That is vision evidence, though, so we run EWC directly. Function-space regularization along the same modes is a natural alternative to replay selection. We include it as a secondary arm because it needs the same probe Jacobians our method computes.

Why not just use adapters and skip continual pre-training? A Comparative Study of PEFT Methods for Python Code Generation finds that domain pre-training predicts downstream performance more strongly than the choice of fine-tuning method. We therefore treat adapters as out of scope for acquiring foundational domain knowledge.

Proposed Experiments

The first decisive check is the forgetting spectrum. We will report the fraction of $\|F\|^2$ captured by the top 1, 6, and 20 modes at 1B and 3B, and whether those modes are stable across seeds, measured as subspace overlap between seeds. We will also report the cosine similarity between predicted and observed one-step probe changes. For reference, the vision paper reports above 0.99, but only on frozen backbones; the file contains no language-model result.

The second check is the main comparison. Medical capability is measured by average accuracy on PubMedQA, MedQA-4-Option, MedMCQA, and MMLU-Medical, plus HOC and DDI-2013. General retention is measured on held-out benchmarks that never appear in $P$, including MMLU (non-medical) and the remaining MATES-suite tasks. The stability gap is measured as the area under the general-accuracy dip curve. All results are reported as replay-budget curves at 1%, 3%, 10%, and 25% of tokens, over three seeds with confidence intervals.

The third check guards against overfitting to the probes. If gains appear only on probe-overlapping tasks and vanish on held-out ones, the method is fitting its own probes rather than protecting general capability, and we will report it as such.