When a language model is trained on text written over many years and then used on text from the future, adding more old text helps less than adding more new text. Everyone who builds corpora from multi-year web crawls or news archives faces this. Nobody has measured the exchange rate: how many old tokens are worth one fresh token, and whether that rate is a single number at all. This proposal argues that it is not. Earlier work found that a stale model's errors on future text pile up in proper nouns and numbers, the words that carry facts, while grammar holds steady (Mind the Gap, Pitfalls of Static Language Modelling). If that is right, a ten-year-old document should count almost fully for learning how language works and very little, or even negatively, for learning what is true now.
We propose to fit a data scaling law that splits the model's loss on future text by token type. The law gives each type its own discount on old data. We then test the practical consequence. A corpus builder should not down-weight whole old documents. It should keep them and stop training on their fact-carrying tokens. If the discounts really differ by type, this token-level treatment should beat the best tuned document-level age discount at equal compute. If they do not differ, the simpler document-level discount is the right tool, and the study will have shown that cleanly.
Core Claim
A scaling law says that test loss falls as a power of the amount of training data. Let $c$ index a token class, either "volatile" (proper nouns, numbers, and other tokens whose correct prediction drifts with time) or "stable" (everything else). Let $D_r$ be the number of recent tokens, $D_h$ the number of historical tokens, and $\Delta t$ the gap in years between the historical data and the evaluation period. We hypothesize that loss on class $c$ in future text follows
$$L_c(D_r, D_h, \Delta t) = E_c + A_c\,\big(D_r + \eta_c(\Delta t)\,D_h\big)^{-\alpha_c} + B_c(\Delta t),$$
where $E_c$ is the irreducible loss, $A_c$ and $\alpha_c$ are the usual power-law scale and exponent, $\eta_c(\Delta t)$ is the exchange rate (how much one historical token counts relative to one recent token), and $B_c(\Delta t)$ is a floor of stale-knowledge error that no amount of old data removes. The form extends the result in Scaling Laws are Redundancy Laws. That paper uses kernel regression to show that temporal correlation in a stationary process only shrinks the effective sample size, shifting the learning curve without changing the exponent. That corresponds to the $\eta$ term with $B = 0$. Drift in what is true should instead create a floor. The specific, falsifiable prediction has two parts. For stable tokens, $\eta \approx 1$ and $B \approx 0$ even at large $\Delta t$. For volatile tokens, $\eta$ falls with $\Delta t$ and $B$ rises with it. A single, class-blind discount is the null hypothesis we are trying to reject.
The idea is closest to three lines of work, and it differs from each. Lazaridou et al. established that decay is roughly linear in time, concentrated in entities and numbers, and not slowed by a larger model (287M to 448M parameters). They also found that a smaller fresh model beats a 60% larger stale one. They varied model size and used dynamic evaluation, but never varied the volume and age of training data, so they give no exchange rate. That work motivates ours and supplies the token-class split. FineWeb found across 96 Common Crawl snapshots from 2013 to 2024 that deduplicating globally hurt performance. Global deduplication kept the oldest copy of long-lived, high-quality pages and left newer snapshots full of transient junk. FineWeb treats time as a nuisance for deduplication and does not model relevance, but its finding warns that crude, document-level temporal operations destroy durable content. That warning constrains our design. Capturing Conversion Rate Fluctuation during Sales Promotions reuses historical data through retrieval and importance weighting, raising offline AUC from 0.793 to 0.875 on Alibaba's 9.9 sales. It relies on shifts that recur every year and weights whole examples, whereas language drift is open-ended and, we argue, localized within examples.
Research Program
The first part is measurement. We train small language models on time-stratified news, varying recent volume, historical volume, and historical vintage on a grid. We evaluate each model on held-out articles from a later year. We split per-token loss by class and fit the equation above separately per class, with $\eta$, $B$, $\alpha$ and $A$ free. We then test whether a shared-$\eta$, zero-floor model fits the class-split curves as well; a likelihood or held-out-error comparison decides this.
There are two ways to define "volatile," and we compare them. The cheap version uses a part-of-speech and named-entity tagger to mark proper nouns and numbers, following Lazaridou et al. The data-driven version assigns each word type a drift score. The drift score is the loss gap on that type between a small model trained on an old vintage and one trained on a recent vintage, both evaluated on recent held-out text. The data-driven version can catch shifted common words and emerging terms that no tagger flags. The data-driven score also addresses a tension raised by Beyond neural scaling laws. That paper shows that keeping hard, atypical examples beats power-law scaling when data is abundant. In a drifting stream, though, obsolete facts also look hard. We will measure how strongly an example-difficulty score correlates with our drift score on old documents. A strong correlation would mean difficulty-based pruning silently favors stale text.
The second part is the intervention the law implies. The intervention is token-level age masking: keep old documents in the training mix, but drop the training loss on their volatile tokens, with the degree of down-weighting set by the fitted $\eta_{\text{volatile}}(\Delta t)$. Those tokens still appear as context, so syntax and discourse structure are still learned from them. The model is simply not trained to predict stale facts. This is a data-side intervention, which matters because Temporal Generalization: A Reality Check showed that manipulating parameters after training fails. In that study, extrapolating the direction weights moved between checkpoints lost to simply deploying the most recent model.
Training order is a confound. Capturing the Temporal Dependence of Training Data Influence finds that a sample's influence depends on when in training it is seen, following a U-shape. We therefore shuffle vintages uniformly in all main runs and treat ordering as a later study.
Baselines
The naive reference points are two. The first is recent-only training, meaning only the latest slice, which is the "Recent Model" that beat every extrapolation method in the reality-check study. The second is pooled uniform training on all vintages, which is standard practice. The paper we build on reports a number for the naive gap. Lazaridou et al. report that Transformer-XL is up to 16% worse in perplexity two years past its training cutoff on CustomNews. They also report that dynamic evaluation cuts perplexity on emerging words by 39.6%. Our pooled baseline should reproduce the qualitative, roughly linear decay on WMT News Crawl before any other comparison is trusted.
The strongest competitor is document-level exponential age discounting, which weights each old document's whole loss by $e^{-\lambda \Delta t}$. We also run its discrete cousin, a tuned mixture ratio over vintages. These are what a practitioner would try first, and they directly compete with the claim that token-level treatment matters. No reported result was found on Lacuna for exponential age-weighting of pretraining data. We will tune $\lambda$ and the mixture ratios with the same budget of trial runs, selected on a validation year, as our method's single scaling knob. All methods see the same documents, the same total tokens, and the same number of optimizer steps.
We also run dynamic evaluation, in the style of Lazaridou et al., stacked on each method. This checks whether our gain survives test-time adaptation or merely duplicates it.
Two ablations isolate the key ingredient. The first masks the same fraction of tokens in old documents, chosen at random rather than by class, to separate "less loss on old text" from "less loss on the right tokens." The second applies volatile-token masking to recent documents as well, which should hurt; if it does not, the effect is just a regularizer. Beating the strongest baseline means lower future-year perplexity on volatile tokens with stable-token perplexity no worse than pooled training, at matched compute. The margin must exceed the spread across three seeds, and it must hold at two model sizes.
Why not just…?
Why not just train on the most recent slice? Because it throws away the stable signal, which our law predicts is nearly full-value regardless of age. FineWeb needed trillions of tokens from many crawls to build a competitive corpus, and a single recent crawl cannot supply that volume. Still, recent-only training is a hard baseline in the reality-check study, and we include it at every grid point. The fitted $\eta_{\text{stable}}$ directly quantifies what it forfeits. If $\eta_{\text{stable}}$ turns out small, recent-only wins and our proposal is wrong.
Why not just down-weight old documents with a tuned decay factor? This is the real threat. If old documents are uniformly less useful, a tuned $\lambda$ captures everything and token masking adds complexity for nothing. The evidence against it is indirect. Decay concentrates in entities and numbers, and FineWeb shows that document-level temporal operations remove durable high-quality text. Neither result proves our intervention is better. The head-to-head comparison at matched tuning budget settles it, and so does the class-split fit: if $\eta$ does not differ between classes, we will say that the simple decay is sufficient.
Why not just use a bigger model? Lazaridou et al. found that going from 287M to 448M parameters lowered perplexity but left the rate of decay unchanged. We will check that this holds at our two model sizes. We will also check whether $\eta$ and $B$ depend on model size, which would itself be a new finding about aging data.
Why not just adapt at test time with gradient steps on incoming text? Dynamic evaluation helps on new words but causes forgetting, and it needs a stream of fresh text at deployment. We treat it as complementary and stack it on every method. If it erases our advantage, the proposal reduces to a pretraining-cost story rather than an accuracy one.
Proposed Experiments
The measurement grid uses WMT News Crawl (2007–2019), with the arXiv abstracts corpus (1986–2019) as a second domain with fast-moving terminology. For a target year $T$, we vary recent tokens (years $T-1$ to $T$) and historical tokens from single vintages at gaps of 2, 5 and 10 years. Each axis gets about five log-spaced volumes, and we train models at roughly 100M and 400M parameters. We evaluate on year $T+1$, reporting perplexity overall and by token class, and loss on emerging words, defined as rare in training and frequent at evaluation.
The first decisive check is the fit. Do volatile and stable tokens need different $\eta$ and $B$? And does the law, fit on smaller $D_h$, predict loss at the largest $D_h$ within seed noise? The second is the intervention comparison against tuned document-level decay, described above. The third is transfer to web scale. Using per-snapshot-deduplicated FineWeb snapshots, we train roughly 1B-parameter models with and without token-level age masking. We measure perplexity on the latest held-out snapshot and check that MMLU and ARC do not fall. FineWeb reports 33% MMLU and 46% ARC for full FineWeb in its ablation setup. We treat those as a sanity reference, not a target, since our token budgets differ.
A falsifying outcome would be any of the following: equal $\eta$ across classes, no floor $B$ for volatile tokens, or a matched-budget loss to document-level decay. Any of these is worth reporting. A positive outcome would give corpus builders a measured price for old data and a concrete recipe for spending it.
References
- Mind the Gap: Assessing Temporal Generalization in Neural Language Models
- Pitfalls of Static Language Modelling
- Scaling Laws are Redundancy Laws
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach
- Beyond neural scaling laws: beating power law scaling via data pruning
- Temporal Generalization: A Reality Check
- Capturing the Temporal Dependence of Training Data Influence