A language model's knowledge is frozen at the date its training data ends. When a user later asks who holds an office or what a guideline recommends, the model usually answers confidently with whatever was true at training time. The standard fix is to retrieve fresh documents, but retrieving on every query is expensive, and retrieval often misses. When it misses, models fall back on their stale memory. A cheaper and more reliable trigger would answer one question: is this particular answer likely to have changed since the model was trained? This proposal uses the model itself to make that call. We sample the model's answer to the same question framed at several different past dates ("as of 2012", "as of 2016", …). If the model's own recollection of the answer shifts across those dates, the fact is one the model knows to be volatile, and the answer it gives for "now" should be treated as probably expired.
The idea turns a known weakness of self-consistency decoding into a signal. Self-consistency decoding samples many answers and takes a majority vote. Its premise is that "there are many ways to be wrong, but typically only one way to be correct" (Solving Quantitative Reasoning Problems with Language Models). That premise breaks for outdated facts, because every sample agrees on the same memorized, obsolete answer. Agreement among repeated samples therefore cannot detect staleness. Disagreement across time-anchored samples might. A human-in-the-loop pipeline, in which the model drafts and people review, supplies what this needs: a verified set of answer timelines for evaluation and calibration.
Core Claim
Our central claim is that spending a fixed sampling budget across time-shifted versions of a question separates expired facts from stable ones better than spending that budget on repeats of one question. Two quantities are compared. The first is cross-date disagreement: how much the model's answers vary across past-date framings. The second is within-date disagreement: how much its answers vary across repeats of a single framing. We define the volatility score
$$V(x) = D_{\text{across}}(x) - \bar{D}_{\text{within}}(x),$$
where $D_{\text{across}}(x)$ is one minus the share of the plurality answer when all samples are pooled over past-date framings of question $x$, and $\bar{D}_{\text{within}}(x)$ is the same quantity computed separately for each framing and then averaged. Subtracting the within-date term is essential. SituatedQA shows that adding temporal modifiers makes models "semantically fragile," so some cross-date variation is plain noise. Only variation beyond that noise counts as evidence that the fact changes over time. We then use $V$ in two ways. It decides when to retrieve. It also decides when to abstain: if the score is high and the retrieved evidence contains no answer, the system declines instead of returning the parametric majority. This rule targets the "outdated trap" reported in RealTime QA.
The claim can fail in a specific, informative way. The model's memory of past changes may not predict whether a fact changed after the cutoff. If so, $V$ will be no better than a fact-duration classifier or plain vote confidence, and the experiments below are designed to show that outcome cleanly.
Relation to Prior Work
The closest work is Mitigating Temporal Misalignment by Discarding Outdated Facts. It trains a separate BERT model to predict how long a fact lasts, compares that prediction to the gap between query time and training time, lowers confidence in facts likely to have expired, and retrieves only then. We keep its adaptive-retrieval setup and its evaluation on SituatedQA. We replace the external duration model with a signal read from the generator's own sampled answers. That paper also notes that it cannot identify when a fact began. Our timelines provide change points directly.
Self-consistency work supplies the sampling machinery but has been tested on static tasks. Confidence Improves Self-Consistency in LLMs weights each vote by the model's own judgment that its answer is true. Integrative Decoding sums next-token scores across sampled paths to improve factuality in long-form text. Both reward agreement, and on stale facts agreement is exactly the failure. The file's analysis of knowledge conflicts suggests that integrating paths this way could even strengthen outdated memory.
DIPPER varies prompts to break biases shared by all samples of one prompt. Our date framings are a structured form of such prompt variation, and we test whether the temporal structure matters beyond generic paraphrase. Estimating the Self-Consistency of LLMs derives how to split a fixed budget between the number of distinct prompts and repeats per prompt. We reuse that result to set how many date framings to use and how many samples to draw for each.
On data, LiveNewsBench uses self-consistency only as an offline filter when building its benchmark. It also excludes facts that keep evolving, which are the facts we target. ToolQA and Building a Role Specified Open-Domain Dialogue System establish the author–reviewer pattern: people design templates or review model drafts, and the dialogue paper reports review running 13.3 times faster than writing from scratch. Both produce data frozen at a single moment. We extend the pattern to time-indexed answers.
Research Program
The first component is a human-reviewed set of answer timelines, which we call DiachronicQA. We start from the temporally dependent questions in SituatedQA and StreamingQA. For each question, a model drafts a timeline of the form (answer, start date, end date) from dated sources. Reviewers then verify or correct each change point instead of writing timelines from scratch. They also flag questions whose answer changed after the backbone model's training cutoff. Those flags form the positive class for detection. Reviewers additionally mark "retrieval-insufficient" cases, where a fixed web snapshot does not contain the current answer. These cases let us measure the outdated trap directly. We report agreement between reviewers and the time spent per item, compared against writing timelines from scratch on a subsample.
The second component is the time-anchored sampling procedure itself. For a total budget $B$ of model calls per question, we set the number of date framings to about $\sqrt{\pi B/8}$ and the samples per framing to about $\sqrt{8B/\pi}$, following Estimating the Self-Consistency of LLMs. With $B=40$ this gives roughly four framings of about ten samples each. The framings are spaced evenly across years before the cutoff. Answers are grouped by normalized exact match plus alias lists taken from the timelines. We also run a variant in which each vote is weighted by the model's self-rated confidence, as in Confidence Improves Self-Consistency in LLMs.
The third component is the decision rule. The system retrieves when $V$ exceeds a threshold. It abstains when $V$ is high and a reader check finds no answer in the retrieved evidence. Otherwise it returns the parametric plurality answer. Thresholds are tuned on a held-out split of DiachronicQA only.
The fourth component is analysis. We will map where the model's remembered past changes diverge from reviewer timelines. We will also test whether $V$ remains informative for rare entities, which SituatedQA identifies as a separate weak point.
Baselines
The naive reference points sit at the two ends of the retrieval-cost trade-off: always answer from memory with greedy decoding, or always retrieve. On RealTime QA, GPT-3 closed-book reaches 44.9% multiple-choice accuracy, and GPT-3 with Google search reaches 66.5% (RealTime QA). On open-ended generation the corresponding figures are 13.3 exact match (EM) with a Wikipedia retriever and 34.6 EM with Google search. We rerun both ends with our backbone and our retriever, so that every method shares one retrieval-cost curve.
The strongest existing method is the fact-duration approach of Mitigating Temporal Misalignment by Discarding Outdated Facts. On SituatedQA it reports 20.5 EM while retrieving for 45% of queries, and a 50–60% reduction in expected calibration error (ECE). ECE measures the gap between the model's confidence and its actual accuracy. This method is the most direct competitor because it solves the same problem, deciding which parametric answers have expired, at a retrieval cost comparable to ours. We retrain its duration model on TimeQA and TimePre as described in that paper. It uses the same backbone reader and retriever as our method, with the same threshold-tuning split. It is allowed a matched budget of greedy calls, or its native single call if extra calls do not help it. Beating it means higher EM at a 45% retrieval rate or lower, plus lower ECE, on the SituatedQA temporal split. We also combine its duration estimate with $V$, to test whether the two signals are complementary.
The self-consistency competitors use the same budget $B$ as our method, spent entirely on repeats of the undated question. The first triggers retrieval when the plurality share falls below a threshold. The second applies the confidence-weighted variant (Confidence Improves Self-Consistency in LLMs). The file reports results for that method only on math and reasoning benchmarks: it reaches a target accuracy with 41–46% fewer samples, and with Gemma2-9B on MATH it beats 30-sample standard voting using 8 samples. No reported result was found on Lacuna for either self-consistency method as a staleness detector.
Two ablations isolate the key ingredient. The first spends the budget on non-temporal paraphrases in the style of DIPPER, with the same number of prompts and samples, and computes $V$ the same way. The second drops the within-date correction term from $V$. If the paraphrase ablation matches the date-anchored version, the gain comes from prompt diversity in general, and the temporal framing adds nothing.
Why not just…?
Why not just use vote agreement from ordinary self-consistency to trigger retrieval? This is the most serious threat, and the one the design targets. Our hypothesis is that stale facts produce confident, unanimous votes, so the vote share stays high exactly when the answer has expired. The detection experiment settles this directly. On reviewer-flagged changed facts, we compare how well repeat-sample agreement and $V$ separate changed from unchanged facts. If agreement does as well as $V$, the proposal reduces to existing self-consistency, and we will report that.
Why not just tell the model today's date? RealTime QA found that adding the current date hurt closed-book accuracy, because dates past the cutoff are unfamiliar inputs to the model. We avoid this by using only past dates, which fall inside the training period. We still include a current-date prompt as an additional baseline.
Why not just always retrieve? Retrieval costs money on every query. It also fails in about a third of errors (34% in RealTime QA), and even with an updated corpus, SituatedQA found accuracy on updated facts 15 points below stable facts. Always retrieving therefore does not remove the fallback to stale memory. Our abstention rule is evaluated on exactly the retrieval-insufficient items that reviewers flag.
Why not just ask the model whether this fact might have changed? Asking the model to state its uncertainty is cheap. However, binary abstention leads to heavy over-abstention on stable facts (Mitigating Temporal Misalignment). We include the model's self-rated confidence as a vote weight and as a standalone baseline.
Proposed Experiments
The first test is detection. On DiachronicQA, built on the SituatedQA and StreamingQA question pools, we measure how well each signal ranks post-cutoff changed facts above stable ones, using the area under the ROC curve, for budgets $B \in \{8, 20, 40\}$. The prediction is that $V$ beats both repeat-sample agreement and the duration model. The paraphrase ablation should fall between them.
The second test is end-to-end question answering. On SituatedQA temporal, we trace EM against retrieval rate and report ECE. On RealTime QA, we report accuracy and EM under a frozen snapshot, to probe the outdated trap. A decisive result would be beating 20.5 EM at 45% retrieval or less.
The third test covers retrieval-insufficient items specifically. We compare how often each system outputs the obsolete answer versus abstaining, predicting that plain voting increases obsolete outputs and $V$-gated abstention reduces them.
The fourth test concerns timing. On StreamingQA, we check whether $V$ computed before a fact changes predicts that change, using the benchmark's time-stamped questions.
All methods use one open backbone of about 8–9B parameters, the same retriever, and matched call budgets. Every threshold is tuned on a single held-out split.
References
- Solving Quantitative Reasoning Problems with Language Models
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- RealTime QA: What's the Answer Right Now?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Confidence Improves Self-Consistency in LLMs
- Integrative Decoding: Improving Factuality via Implicit Self-consistency
- Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks
- Estimating the Self-Consistency of LLMs
- LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News
- ToolQA: A Dataset for LLM Question Answering with External Tools
- Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models