When a language model is updated with new text, much of that text is already out of date. A 2019 article naming a country's prime minister is a correct record of 2019. Read in 2021, it is a wrong answer. Continual training on time-stamped corpora therefore feeds the model a mix of current facts, durable facts, and expired facts, all written with the same confident prose. We propose to treat this as a label-noise problem. Each document's claim about a fact is a noisy label for the fact's value at query time. The probability that the label is wrong can be estimated from two things: how old the document is, and how long facts of that kind usually last. With that probability in hand, well-studied corrections from the noisy-label literature can be applied to knowledge updating. Chief among them is forward loss correction, which scores the model against the corrupted label distribution it should expect to see instead of trusting each target as ground truth.
The idea is not simply to down-weight old documents. Under forward correction, how much a stale document pulls on the model depends on what the model currently believes. If the model already predicts a newer value, an old document asserting the previous value is mostly discounted. If the model has not yet learned the newer value, the same document is trusted. This creates a chicken-and-egg problem that curriculum learning is suited to solve: the model must first see low-risk evidence, meaning fresh documents and durable facts, so that its beliefs can correctly discount risky old documents later. Evaluation is designed around contamination. Test facts must lie after the base model's training cutoff and must not be answerable by recalling memorized benchmark strings.
Core Claim
Consider a query time $t_q$, a training document written at time $t_e$, and a fact type with predicted lifespan distribution $F$. The document's age is $a = t_q - t_e$. The probability that its asserted value has since expired is
$$\rho = F(a),$$
the cumulative probability that a fact of this type lasts less than $a$. This is the quantity that Mitigating Temporal Misalignment by Discarding Outdated Facts computes at inference time to shrink a QA system's confidence. That work predicts fact duration with a BERT cloze model and compares it against the evidence-query gap. We move the same quantity into the training loss.
Let $\tilde y$ be the value the document asserts and $p_\theta(\tilde y \mid x)$ the model's probability of that value for the current state of the world. The corrected loss is
$$\mathcal{L}(\theta) = -\log\big[(1-\rho)\,p_\theta(\tilde y \mid x) + \rho\,u(\tilde y \mid x)\big],$$
where $u$ is a fixed distribution over stale values. $u$ puts mass on values that other, older documents assert for the same fact, plus a small uniform floor. This is the forward correction of Making Neural Networks Robust to Label Noise: a Loss Correction Approach, specialized to an open answer space with a two-component noise model.
Its gradient with respect to $\log p_\theta$ is the ordinary maximum-likelihood gradient multiplied by
$$w = \frac{(1-\rho)\,p_\theta}{(1-\rho)\,p_\theta + \rho\, u}.$$
This weight is the posterior probability that the document is still correct, given the model's current belief. A fixed recency weight would be $1-\rho$ regardless of the model. Here $w$ adapts: the more the model already disagrees with a likely-stale document, the less that document pushes on it.
The claim has two parts. First, belief-dependent correction should reduce the rate at which updated models give outdated answers, compared with filtering or fixed recency weighting, without losing durable facts that appear only in old documents. Second, the correction works only if it is paired with a fresh-first curriculum. Otherwise the posterior $w$ inherits the model's stale beliefs and ends up protecting them.
Why this is the right intersection
Three findings motivate the framing. SituatedQA: Incorporating Extra-Linguistic Contexts into QA re-annotates questions across time and finds that 16.5% change answers. Even with an updated retrieval corpus, models were 15 percentage points less accurate on updated facts than on stable ones, so new text alone does not override old beliefs. RealTime QA: What's the Answer Right Now? shows that models fall back on obsolete parametric answers when retrieval is weak, and 34% of its system errors came from retrieval failures. TimeLMs: Diachronic Language Models from Twitter continually pretrains on quarterly slices in plain chronological order, which is the naive update our method would replace. It never asks which ingested claims are already expired.
Two results constrain the design. Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification finds pretrained models robust to random injected noise: accuracy fell only about 10% at 70% noise on AG-News. Feature-dependent weak-supervision noise, however, caused a 35% drop on Hausa, and specialized noise methods gained under 1.5% over early stopping on a noisy validation set. Stale facts are feature-dependent noise in exactly this sense: the document text is correlated with the wrong value. That result is both our motivation and our strongest warning. Generic loss-based noise handling may do nothing here, which is why we inject external information ($\rho$ from dates and fact types) rather than relying on loss statistics alone. Second, Making Deep Neural Networks Robust to Label Noise: Cross-Training With a Novel Loss Function shows that a single model selecting its own training samples reinforces its early errors. Our posterior $w$ is a form of self-selection and is exposed to the same risk. The curriculum is our answer to that risk.
The closest method is Learning under Temporal Label Noise. It introduces a Forward Temporal Loss with a time-varying noise matrix $Q(t)$ and cuts error on the moving sensor dataset from a 14.9% best static baseline to 4.2%. It assumes, however, that the relationship between inputs and true labels is fixed and that one noise function applies to all streams, and it is evaluated only on low-dimensional sensor series. In our setting the true label itself changes, and the noise rate depends on both document age and fact type, since a head-of-state fact and a birthplace fact decay at very different rates.
The curriculum's role draws on A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula. That paper proves easy-to-hard ordering helps only under "feasible initialization," meaning the model must already succeed on the first stage. It also documents a regression when this fails (Llama-3.2-1B, −0.28% on GSM8K). Its theory assumes a clean verifier. Ours is a noisy one, $w$, whose reliability depends on what was learned earlier. "Easy" here means low $\rho$, in the spirit of self-paced learning (Self-Paced Learning for Latent Variable Models), which admits low-loss samples first. We order by an external staleness score rather than by the model's own loss, and that difference is what separates our curriculum from confirmation-biased self-selection.
Research Program
The first component is a controlled synthetic testbed that is contamination-free by construction, in the spirit of the generated tasks of DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. We generate fictional entities with attributes of known lifespan distributions, ranging from volatile roles to near-permanent attributes. Each attribute follows a simulated timeline of value changes. Documents are emitted at random times and assert the value current when they were written, using varied surface templates to limit the structural predictability noted for INVITE.
Because entities are fictional, the base model has no prior, and the generator knows the true $\rho$ for every document. We can sweep the staleness rate the way the noisy-label literature sweeps injected noise. We can also separate two error sources: errors from the loss itself, measured with an oracle $\rho$, and errors from a mis-estimated $\rho$, measured with the learned duration predictor.
The second component is a real-world setting. We take an open base model whose pretraining cutoff predates SituatedQA's later annotation period, set $t_q$ to that period, and continually train on a mixed-age corpus. The corpus combines passages from the older Wikipedia snapshot used by SituatedQA, the newer snapshot, and time-stamped news from the RealTime QA archive. Each training assertion receives a timestamp and a fact type.
$\rho$ comes from a duration predictor trained, as in Zhang and Choi, on TimeQA and TimePre with distant supervision. That paper reports its classifier placed 65% of temporally dependent facts within a three-year window, so $\rho$ will be noisy. We therefore also test a softened loss that tempers $\rho$ toward 0.5 in proportion to predictor uncertainty.
The curriculum admits documents in increasing order of $\rho$ in three stages. The stage thresholds are chosen on the synthetic testbed and then frozen before real-world runs. The stale-value distribution $u$ is built by grouping documents that assert values for the same entity-relation pair, which requires entity linking. Errors in that grouping are a measured failure mode, not an assumption.
Baselines
The naive reference point is plain continual fine-tuning on the full mixed-age corpus in shuffled order with standard maximum-likelihood loss, the analogue of the TimeLMs recipe. Every other method must at least match its accuracy on stable facts while beating it on updated ones.
The strongest generic competitor is early stopping on a noisy validation set, because Is BERT Robust to Label Noise? found it matched or beat specialized noise methods on feature-dependent noise. It gets the same data and a validation split drawn from the same mixed-age distribution, checkpointed at the same frequency.
The strongest temporal-noise method is the Forward Temporal Loss of Learning under Temporal Label Noise, with $Q(t)$ depending only on document age and estimated by its plug-in method. This isolates the value of conditioning on fact type. No reported result was found on Lacuna for this method on any language or QA benchmark; its only language-free reported number is the 4.2% moving error.
The strongest system-level competitor is inference-time handling of obsolescence. We run the duration-based confidence discounting with adaptive retrieval of Mitigating Temporal Misalignment by Discarding Outdated Facts, which reports 20.5 exact match (EM) on its 322-question SituatedQA slice while retrieving on 45% of queries, and a 50–60% reduction in calibration error. We run it on the same base model and evaluate on that slice as well as our larger split, so its numbers are comparable. We also run open-book retrieval over the newest corpus. As a reference, RealTime QA reports 66.5% multiple-choice accuracy and 34.6 EM for GPT-3 with Google Search, versus 44.9% closed-book. Our comparison uses our own base model, not those numbers directly.
Three ablations isolate the proposal's ingredients:
- Recency filtering drops documents with $\rho$ above a tuned threshold.
- Static importance weighting trains with fixed weight $1-\rho$, using the same $\rho$ as our loss. This isolates the belief-dependent posterior.
- Shuffled forward correction uses our loss without the fresh-first ordering. This isolates the curriculum.
All methods get the same corpus, token budget, and base model. Each gets an equal hyperparameter search of the same number of trials, tuned on a held-out validation set of synthetic entities and a disjoint SituatedQA development split.
Beating the baselines means two things. The method must lower the stale-answer rate on updated facts beyond the best baseline by more than seed-to-seed variation across three seeds. It must do so while staying within one EM point of the naive baseline on stable facts, so that gains do not come from forgetting durable knowledge.
Why not just…?
Why not just train only on recent documents? Recency filtering is the cheapest fix and will likely handle volatile facts well. It fails when durable facts appear mainly in old text, and the SituatedQA split lets us measure that directly on stable facts. The synthetic testbed makes the trade-off explicit: as the proportion of durable facts whose only mentions are old grows, filtering must lose stable-fact accuracy while forward correction should not. If filtering matches our method on both updated and stable facts in the real corpus, we will report that the duration information adds nothing beyond a date cutoff.
Why not just weight documents by $1-\rho$? This is the real threat. If fixed weighting matches the belief-dependent loss, the contribution shrinks to "use Zhang and Choi's duration CDF as training weights." That would still be useful, but it is a smaller claim. Our argument is that fixed weighting still applies substantial gradient toward the stale value whenever $\rho$ is moderate, whereas the posterior shuts that gradient off once the model has learned the newer value. The static-weighting ablation, run at the same $\rho$, settles this. We expect the gap to be largest at intermediate staleness rates in the synthetic sweep. If it does not appear there, we drop the claim.
Why not just use retrieval and leave the model frozen? Retrieval over current text is complementary, not a substitute. SituatedQA shows a 15-point deficit on updated facts even with an updated corpus, and RealTime QA shows models fall back on stale parametric answers when retrieval misses. We test the combination directly: retrieval over a model updated with our method versus retrieval over a naively updated model. The key measurement is the stale-answer rate on queries where retrieval returns nothing relevant, which is exactly where the "outdated trap" appears.
Why not just stop early? For feature-dependent noise, early stopping has been as good as anything tried, per Zhu et al. The reason to expect it to fall short here is that stale documents are not memorized late: they agree with the model's own pretraining prior and are fit from the first step. If the early-stopping checkpoint equals our stale-answer rate, that mechanism is wrong. The training curves of stale-answer rate over steps will show which is the case.
Proposed Experiments
The first decisive check is on the synthetic testbed with oracle $\rho$. We sweep the staleness proportion and fact-lifespan mix, and plot stale-answer rate and durable-fact accuracy for every method. The prediction is that belief-dependent correction beats static weighting at intermediate staleness. We also predict that shuffled correction is worse than curriculum correction, most visibly when stale documents outnumber fresh ones for a fact, which is the regime where confirmation bias should bite. We then replace oracle $\rho$ with predicted $\rho$ and report how much of the advantage survives. This quantifies how sensitive the method is to the duration predictor's accuracy.
The second check is real-world. We evaluate on the SituatedQA temporal split, reporting EM separately on updated and stable facts, plus the fraction of answers matching the older annotated answer (the stale-answer rate). We also report calibration error for comparison with Zhang and Choi, and closed-book accuracy on the RealTime QA real-time questions whose dates fall within the corpus window.
To keep the evaluation free of contamination, we run Min-K% Prob membership inference, following Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?, on test questions and answer strings against the base model. We discard items it flags as likely seen. Test questions are paraphrased so no training passage contains them verbatim, which separates genuine updating from recall of a leaked question string.
The final analysis examines where the method fails: per fact type, per duration-predictor error, and per entity-linking error in building $u$. Together these should show whether treating obsolescence as label noise is the right mechanism, or whether a timestamp filter captures most of the benefit.
References
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- RealTime QA: What's the Answer Right Now?
- TimeLMs: Diachronic Language Models from Twitter
- Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification
- Making Deep Neural Networks Robust to Label Noise: Cross-Training With a Novel Loss Function
- Learning under Temporal Label Noise
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
- Self-Paced Learning for Latent Variable Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?