Calibrated Loss Caps Against Memorization in Domain Pre-Training

When a general language model is further trained on a small specialized corpus such as financial news, clinical abstracts, or high-performance computing code, it learns the domain, but it also starts reciting its training documents word for word. This second effect is the language-model version of a familiar failure in image generation, where a generator trained on ten or a hundred pictures collapses into copying them. We propose a simple, targeted fix. During training, each token's improvement under the adapting model is compared against the improvement that same kind of token shows on held-out domain text. Once a training token has improved more than held-out text ever does, it stops contributing gradient. The model can keep learning whatever generalizes across the domain, including recurring boilerplate, and is blocked from learning what is unique to one document.

The motivation comes from a recent audit. Quantifying Memorization in Continual Pre-training with Japanese General or Industry-Specific Corpora continued training Llama 3 with LoRA, a method that trains only small low-rank weight updates, on deduplicated Nikkei financial news. Verbatim memorized prefixes grew from 15 to 27 characters, against 1.34 to 1.53 on Wikipedia. Membership inference attacks, which guess whether a document was in the training set, reached an AUC of 0.689. The audit proposed no mitigation, and it treated memorized standard disclaimers the same as memorized private specifics. Our claim is that the per-token learning signal itself can separate these two cases, and that doing so lowers leakage at almost no cost to domain accuracy.

Core Claim

The few-shot image-generation literature offers two observations that carry over. First, A Geometric Framework for Understanding Memorization in Generative Models distinguishes two kinds of memorization:

  • Data-driven memorization: the data itself has almost no freedom, for example duplicated or formulaic content. Reproducing it is unavoidable and mostly benign.
  • Overfitting-driven memorization: the model collapses onto individual examples even though the true distribution has more variety.

That paper measures the difference with local intrinsic dimension on image manifolds and leaves discrete text unformulated. Second, Bridging Data Gaps in Diffusion Models with Adversarial Noise-Based Transfer Learning documents an "unbalanced pace": some samples are already memorized while others are still underfit. A single global stopping point or regularization strength is therefore wrong for most samples at once.

Our proposal turns both observations into a per-token, text-native criterion without estimating intrinsic dimension. We define the following quantities:

  • $p_0$ is the frozen base model and $p_\theta$ the adapting model.
  • $x_t$ is a token and $x_{<t}$ its prefix.
  • $g_t = \log p_\theta(x_t\mid x_{<t}) - \log p_0(x_t\mid x_{<t})$ is the token's log-probability gain.
  • $b(t)$ is the token's base-surprisal bin, that is, how predictable the base model already found it.

On a document-disjoint held-out slice of the domain corpus we estimate $q_b$, the $(1-\alpha)$ quantile of gain within each bin $b$. Training then uses

$$ \mathcal{L}(\theta) = -\frac{1}{|T|}\sum_{t\in T} m_t \log p_\theta(x_t\mid x_{<t}), \qquad m_t = \mathbb{1}\left[g_t \le q_{b(t)}\right], $$

where $T$ is the batch's token set. We recompute the quantiles every $K$ steps as the model changes, and we cache base log-probabilities offline. Formulaic domain text recurs across documents, so its held-out gain is high and it keeps being learned. A string specific to one document shows no comparable held-out gain, so its gradient is cut once it exceeds what generalization explains.

This differs from the contrastive source anchoring of A Closer Look at Few-shot Image Generation, which ties every target feature to a frozen source generator. Our method constrains nothing that generalizes. It also differs from Annotations Mitigate Post-Training Mode Collapse, which preserves topic diversity by tagging the pre-training corpus and requires a costly tagging pass. Our method needs no tags and targets leakage rather than topical diversity.

Research Program

We first establish that domain continual pre-training on public corpora behaves like a few-shot regime, and that the regime worsens as the corpus shrinks. This is the language analog of the FFHQ-100 versus FFHQ-1K gap reported by Training Diffusion-based Generative Models with Limited Data, where estimation error dominates when the sample count is small. We sweep corpus size at 10M, 100M and 1B tokens drawn from the Financial News CommonCrawl and SEC filings corpus used by Efficient Continual Pre-training for Building Domain Specific Large Language Models. We repeat the sweep on PubMed abstracts as used by BioBART. The Nikkei corpus itself is proprietary, so we do not use it.

Into each corpus we insert synthetic canaries, meaning fake account numbers and patient identifiers, at controlled repetition counts of 1, 4, 16 and 64. These give ground-truth private content alongside naturally recurring boilerplate.

We train two models:

  • Pythia-6.9B with full parameter updates, following the FinPythia setup, because its pre-training data is public.
  • Llama-3-8B with LoRA, mirroring the Nikkei audit.

The analysis has three parts. We measure how the gated fraction of tokens evolves over training, as a direct readout of unbalanced pace. We test whether gated tokens concentrate on canaries and document-specific named entities rather than formulaic spans. Finally, we test a secondary hypothesis about promptability. Effect of Multilingual and Domain-adapted Continual Pre-training on Few-shot Promptability found that domain continual pre-training hurts few-shot prompting on domain tasks, while later fine-tuning restores it. Part of that loss may come from collapse onto training-document continuations. If so, capping should preserve few-shot BC5CDR named-entity accuracy.

Baselines

The naive reference is plain continual pre-training with standard early stopping, using the same data, tokens and learning-rate sweep as our method. It anchors both axes: how much the domain gains and how much leaks. For our corpora we found no reported memorization result on Lacuna for PubMed or SEC/Financial News continual pre-training. The nearest reported figure is the Nikkei LoRA audit's membership AUC of 0.689 together with its prefix growth from 15 to 27 characters. We will re-run the audit's exact protocol on our corpora, measuring longest verbatim prefix, normalized Levenshtein similarity, and the LOSS, zlib, Min-K%, Min-K%++ and ReCaLL attacks.

On utility, the strongest reported domain result in the file is FinPythia's: an 8.3% average improvement over base Pythia on financial tasks, and 12.45% over the baseline when trained on its curated 10% subset (Efficient Continual Pre-training). For biomedical summarization, BioBART-Large reports a Rouge-L of 53.15 on MeQSum.

We reproduce that curated 10% selection as a utility baseline. Because data selection is orthogonal to capping, we also apply capping on top of it.

The strongest competitor on the leakage question is a KL regularizer toward the frozen base model combined with replay of general data. This is the standard way to keep an adapted model close to its origin, and it is the language analog of DCL's generator anchoring. Every regularized method, including ours, gets the same tuning budget of eight strength settings, so each traces a full curve.

A LoRA arm sweeps rank from 4 to 256. It tests whether a simple capacity limit suffices.

The ablation that isolates the key ingredient is an uncalibrated cap. It zeroes the gradient on any token whose loss falls below a fixed global threshold, with no held-out comparison and no surprisal bins. If this ablation matches our method, then the held-out calibration, which is the central idea, adds nothing.

Beating a baseline means this: at matched domain utility, defined as within one point of plain continual pre-training on the averaged FLARE or biomedical suite, our method achieves lower membership AUC and lower canary extraction than the baseline does at its own matched-utility setting.

Why not just…?

Why not just deduplicate the corpus and stop training earlier? Deduplication removes exact repeats, which addresses only data-driven memorization. The Nikkei corpus was already deduplicated and still showed sharp memorization growth. Global early stopping applies one stopping time to every document, which the unbalanced-pace finding argues against. Our early-stopping baseline and the canary repetition sweep will show whether single-occurrence canaries are extracted before the domain score peaks.

Why not just regularize toward the base model with a KL penalty? This is the real threat. It may reduce leakage as well as our method does. Our prediction is that it pays more utility for the same protection, because it slows learning on every token, including the domain idioms continual pre-training exists to teach. The matched-utility comparison above settles this directly. If the KL curve dominates ours, the proposal fails.

Why not just use low-rank LoRA? The Nikkei audit found that LoRA still leaks on industry text, and the inverse-scaling result in Annotations Mitigate Post-Training Mode Collapse suggests that capacity limits alone do not stop collapse. The rank sweep tests whether any rank reaches our leakage level without the utility loss reported for low-rank updates.

Why not just train with differential privacy? It gives a formal guarantee, which our method does not. We found no reported result on Lacuna for differentially private domain continual pre-training on these corpora. We include one differentially private run at a moderate privacy budget as a reference point, and we frame our method as an inexpensive empirical mitigation rather than a replacement for a formal guarantee.

Decisive Checks

The method succeeds if four things hold:

  1. Size sweep. On the corpus-size sweep, canary extraction under plain continual pre-training rises as the corpus shrinks, while under capping it stays near the never-trained rate for canaries repeated up to 16 times.
  2. Membership attacks. Membership AUC under capping moves toward 0.5 at matched FLARE and MeQSum utility, and does so more than with KL regularization or rank-limited LoRA.
  3. Ablation. The uncalibrated cap is clearly worse on at least one of the two axes.
  4. Promptability. Few-shot BC5CDR accuracy degrades less than under plain continual pre-training.

If capping reduces leakage only by lowering utility along the same curve as KL regularization, the per-token calibration idea is refuted.