When a language model is further trained on a specialized corpus such as financial filings, contracts, or PubMed abstracts, most of the loss it minimizes comes from tokens it could have guessed from the previous few words. These include function words, domain filler like patient or hereto, and whole boilerplate phrases like "now therefore, in consideration of the mutual covenants contained herein." The rare entities, figures, and clause-specific terms that make the domain worth learning contribute only a small share of the gradient.
Topic modelers met this problem twenty years ago. Their fix was a background sink: a cheap distribution that explains the predictable tokens, so that the expensive part of the model only has to explain the rest. This note proposes the same fix for domain continual pre-training. We pair the transformer with a fixed n-gram model (a lookup table of next-token probabilities given the last few tokens) trained on the domain corpus. The two are combined as a mixture, with a learned per-token switch, so each token's training signal is split according to how much the n-gram model already accounts for it. The transformer then keeps every document but spends little gradient on local, predictable text. We test whether this improves learning of rare domain terms and reduces the loss of general ability that heavy continual pre-training is known to cause.
Core Claim
Standard next-token cross-entropy gives every token equal weight. Because domain corpora follow a steep frequency distribution, this lets repetitive local structure dominate adaptation. Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training shows the effect in general web text. Enlarging the vocabulary lowers loss on the top 2,500 words by about 0.1 nats but raises loss on the rarest 20,000 words from 11.183 to 13.399 nats. The headline loss still improves (3.179 to 3.136) because frequent words make up about 75% of tokens.
We claim that a context-dependent background sink fixes this better than filtering documents or reweighting by global frequency. By context-dependent we mean the down-weighting is decided per occurrence by a local n-gram model, not per word type. Under the sink, patient is discounted where it is predictable ("the patient was") but keeps full weight where it is surprising. A multi-token boilerplate span is absorbed as a whole. A rare chemical name inside an otherwise templated document keeps its full signal. None of this needs stop-word lists, part-of-speech taggers, syntax trees, or labeled template spans.
Concretely, let $p_\theta(w_t \mid w_{<t})$ be the transformer, $q(w_t \mid w_{t-n+1:t-1})$ a smoothed $n$-gram model over the same subword tokens trained on the domain corpus, and $\lambda_t = \sigma(g_\phi(h_t)) \in (0,1)$ a switch predicted from the transformer's hidden state $h_t$. We train on
$$ \mathcal{L} = -\sum_t \log\big[\lambda_t\, q(w_t \mid w_{t-n+1:t-1}) + (1-\lambda_t)\, p_\theta(w_t \mid w_{<t})\big] + \gamma\,\big(\bar{\lambda} - \rho\big)^2, $$
where $\bar\lambda$ is the batch-mean switch value, $\rho$ is a target background share, and $\gamma$ sets the strength of that prior. The prior plays the role of the asymmetric Dirichlet in Wallach et al.: without it, the switch could drift toward the transformer on every token as the transformer improves, and the sink would stop doing anything.
The gradient reaching $p_\theta$ on token $t$ is the ordinary cross-entropy gradient scaled by $1 - r_t$, where
$$ r_t = \frac{\lambda_t q_t}{\lambda_t q_t + (1-\lambda_t) p_{\theta,t}} $$
is the posterior probability that the background produced the token. Tokens the n-gram explains well therefore train the transformer weakly. At deployment the mixture is the model, and an n-gram lookup costs almost nothing per token. For fine-tuning on downstream tasks we use the transformer's representations directly.
Where This Sits
The mechanism comes from topic modeling. The file describes Wang's structural topic model, which uses a latent switch between topic and background distributions, and Virtanen's document-level empirical sink. It also describes TopicRNN, where a recurrent network absorbs local syntax so that global topics can hold semantics. Putting Back the Stops is closest in spirit. It routes tokens that a local context network finds predictable into a separate syntactic subspace, which raised 20 Newsgroups classification from 75.2% with manual stop-word removal to 89.7%. It supplies our criterion, local predictability, but works on bag-of-words autoencoders and cannot capture sentence-long boilerplate. We move the criterion into a transformer's training loss and use n-grams of order up to about 7 so that multi-token templates are covered. CAST filters background words by how stable their contextual embeddings are across the corpus. That requires embedding every occurrence and computing pairwise similarities, and the file flags this as a scaling bottleneck. A Kneser–Ney n-gram table scales to billions of tokens.
Four domain pre-training papers motivate and constrain the design. Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining shows that separating legal boilerplate from dynamic slots helps: CUAD-NER F1 rose to 0.7507 from 0.7355 for LEGAL-BERT. It needed curated static and dynamic spans, and we ask whether an unsupervised sink recovers that gain. MonoCoder removes superficial local patterns in code by anonymizing identifiers through syntax-tree parsing, which does not exist for prose. We treat the n-gram sink as a soft, parser-free analogue. Efficient Continual Pre-training for Building Domain Specific Large Language Models finds that the best 10% of a 24B-token financial corpus beats all of it. That shows redundancy hurts, but the method discards whole documents, and we operate inside documents instead. Continual Pre-Training is (not) What You Need in Domain Adaption reports that heavy legal continual pre-training degrades reasoning and instruction following relative to LoRA. We test the hypothesis that part of this damage comes from gradient spent fitting domain boilerplate.
Frequency Is What You Need is the main constraining result. Masking tokens by unigram frequency beat syntax-based masking in image–caption pre-training (39.3% vs 35.0% zero-shot ImageNet). This means global frequency alone is a strong signal, and our method must beat it rather than assume context helps. Pulling the other way, BioBART found that disrupting discourse structure (sentence permutation) hurts biomedical generation. Our method therefore never deletes or reorders text; it only reallocates gradient.
Research Program
We build the sink for decoder models in two regimes. The main regime is continual pre-training of Pythia-1.4B, with a confirmation run at 6.9B to match Efficient Continual Pre-training. It covers two domains: the 24B-token financial corpus of news and SEC filings, and a legal corpus of SEC filings plus LEDGAR provisions. The secondary regime is a BERT-base masked language model trained from scratch on PubMed abstracts with an in-domain vocabulary, following Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. There the background model is a bidirectional n-gram over the neighbors of the masked position.
The n-gram models are trained on the domain corpus, tokenized with the language model's own tokenizer, at orders 3, 5, and 7 with Kneser–Ney smoothing. Background probabilities are precomputed and stored alongside each training shard. For continual pre-training, the switch head is initialized to output $\lambda \approx 0$ and $\rho$ is warmed up over the first 5% of steps, so that training starts as ordinary continual pre-training.
The analysis has three parts. The first breaks held-out domain loss into frequency tiers, as in the vocabulary-imbalance paper, and adds entity-level loss on annotated spans, such as chemicals and diseases in the BLURB NER sets and clause spans in CUAD. The second inspects which spans get high $r_t$, to check that the sink absorbs boilerplate rather than rare but locally predictable technical collocations; this is the main failure mode. The third tracks general-domain retention throughout training.
Baselines
The naive reference is the base model evaluated with no domain training, alongside plain continual pre-training on 100% of the domain corpus with standard cross-entropy. Every other method is compared at the same tokens seen, the same optimizer steps, and the same learning-rate sweep of three values per method. Loss weights in all reweighting methods are normalized to mean one per batch, so that no method gains a larger effective learning rate.
The strongest existing competitor is document-level selection from Efficient Continual Pre-training. Relative to base Pythia-6.9B, the file reports an 8.3% average gain on financial tasks for full-corpus adaptation and a 12.45% gain for the 10% selected subset. The file labels that subset ETS-DACP in one summary and ETA-DACP in another. We reimplement the task-agnostic variant, which scores documents by perplexity under a 70M model and by part-of-speech entropy, and run it at a matched token budget. We count the sink as beating it if it exceeds this method on FLARE at equal tokens. We also run the sink on top of the selected 10% to test whether the two effects add.
On legal data, the reference numbers are LB-PU at 0.7507 CUAD-NER F1 versus 0.7355 for LEGAL-BERT, and DeBERTa-xlarge at 47.8% AUPR on CUAD extractive review. For biomedicine, the reference is PubMedBERT's BLURB score of 81.16. A same-budget reproduction of plain in-domain masked-language-model training serves as the controlled comparison.
Three ablations isolate the key ingredient, which is context-dependent attribution. The first replaces $q$ with a unigram model, making it a frequency-only sink. The second removes the learned switch and weights each token by fixed background surprisal. The third is CLIPF-style frequency masking adapted to next-token loss. If the unigram sink matches the full method, the context claim fails.
Why not just…?
Why not just reweight the loss by inverse unigram frequency, as CLIPF effectively does? This is the most serious threat. Frequency Is What You Need shows global frequency is a strong, cheap signal. However, it penalizes a word type everywhere, including surprising occurrences of domain words like cell or claim, whose ambiguity the file lists as an open problem that frequency cutoffs cannot solve. It also cannot see multi-token boilerplate made of individually mid-frequency words. The unigram-sink ablation tests this directly. We report entity-level loss and CUAD separately, because context-dependent attribution should matter most there.
Why not just select the best documents, as in ETA-DACP? Selection removes redundancy but throws away rare content embedded in templated documents. CUAD illustrates the risk: target clauses sit inside contracts that are 99% non-target text (CUAD). The combination experiment settles this. If selection plus sink equals selection alone, token-level attenuation adds nothing beyond document filtering.
Why not just deduplicate boilerplate n-grams out of the corpus before training? Removing text breaks the discourse structure that BioBART shows generative biomedical models depend on, and it makes the model worse at producing the boilerplate users still need. We include a baseline that removes repeated long n-grams, matched for tokens, and measure both downstream scores and held-out perplexity on unmodified domain text.
Why not just use LoRA and skip continual pre-training? Continual Pre-Training is (not) What You Need reports LoRA-based BLAWSTRAL at 56.54% on legal essay grading, beating heavy continual pre-training. That is a real alternative when general reasoning matters most. We run a LoRA baseline at matched compute on the same corpus. The sink is worthwhile only if it keeps LoRA-level general retention while matching or exceeding full continual pre-training on domain tasks.
Proposed Experiments
The first decisive check is whether rare terms benefit. Relative to plain continual pre-training, the sink should lower loss in the rarest frequency tier and on annotated entity spans. Mixture perplexity on frequent tokens should stay within a small margin, otherwise the rare-term gain has simply been bought with fluency. If the tail does not improve, the core claim is wrong.
The second check is downstream performance. We fine-tune on FLARE for finance; on CUAD (AUPR and precision at 80% recall), LEDGAR clause classification, and MULTI-EURLEX for law; and on BLURB for the from-scratch biomedical model. The sink must beat plain continual pre-training, the unigram ablation, and matched-budget document selection, with differences reported over three fine-tuning seeds.
The third check is retention. We measure MMLU and ARC before and after domain training. The hypothesis linking boilerplate fitting to forgetting predicts smaller drops under the sink. If retention matches plain continual pre-training while domain scores rise, the sink helps learning but does not explain forgetting, and we will report it that way.
The fourth check is a sweep over $\rho$ and n-gram order. Performance should peak at an intermediate background share. Performance should rise with n-gram order on legal text, where templates are long, more than on financial news. That pattern is a falsifiable signature of boilerplate absorption rather than generic loss smoothing.
References
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
- Putting Back the Stops: Integrating Syntax with Neural Topic Models
- CAST: Corpus-Aware Self-similarity Enhanced Topic modelling
- Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- Efficient Continual Pre-training for Building Domain Specific Large Language Models
- Continual Pre-Training is (not) What You Need in Domain Adaption
- Frequency Is What You Need: Word-frequency Masking Benefits Vision-Language Model Pre-training
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review