When a language model answers "Who is the CEO of X?" with the person who held the job three years ago, most uncertainty signals miss it. The model is not hesitating. It is confidently reporting what was true when its training data was collected. Existing tools that treat instability under rephrasing as a warning sign assume that wrong answers are shaky answers. Stale answers break that assumption. This note proposes a different probe. We ask the same question under several explicit time frames ("as of 2015", "as of 2019", "currently", "most recently") and map each answer onto a structured record of which values were true during which intervals. From that mapping we read off the date the model's knowledge implicitly comes from. If the "currently" answer lands on an interval that closed before the query date, the answer is stale, even when every surface paraphrase agrees.
The opportunity matters because the two failures need different fixes and are confused in practice. An uncertain answer calls for abstention. A stale answer calls for retrieval, and specifically retrieval that overrides parametric memory. The probe joins three threads. The first is prompt sensitivity, meaning how much a model's output changes under wording that should not matter. The second is knowledge obsolescence, meaning frozen weights drifting away from the current state of the world. The third is hybrid semantic modeling: combining fuzzy, similarity-based matching of free text with a precise structured record, here a fact's timeline. Reproducibility enters as a constraint on how the probe is built and reported.
Core Claim
We make two claims. The first is that surface prompt sensitivity is a weak detector of temporal staleness, while temporal-frame sensitivity, read against a structured timeline, is a strong one. The second is that a staleness score built this way decides when to retrieve better than the current fact-duration approach, and better than general sensitivity-based abstention.
Three lines of prior work motivate the first claim. On the Relation between Sensitivity and Accuracy in In-Context Learning shows that disagreement across instruction paraphrases and demonstration orders correlates negatively with accuracy (Pearson $r$ up to −0.40). It turns this disagreement into an abstention method, SENSEL, but tests it only on static classification. ProSA traces sensitivity to flat output distributions: when the model is unsure, small prompt changes flip the top token. Rethinking Hallucinations points the other way. It reports that some benchmarks, notably TruthfulQA, are dominated by "prompt-agnostic errors (consistent incorrectness)". It also reports that common hallucination detectors pick up randomness rather than falsehood.
Our hypothesis is that post-cutoff changed facts behave like TruthfulQA, not like the flat-distribution regime ProSA describes. RealTime QA supports this: its models "revert to stale parametric memory" rather than hesitate. If the hypothesis holds, SENSEL-style disagreement will under-flag exactly the answers that most need retrieval.
Research Program
The probe has three parts.
The first part is a set of time-frame perturbations. For each question we fix, before seeing any results, a set of $K$ phrasings that anchor the question at different reference times. The set includes past years spanning the fact's known history, an unanchored "currently" form, and a "most recently" form. SituatedQA found that explicit temporal modifiers confuse models. We use that confusion as the measurement, not as noise.
The second part, the hybrid component, maps each answer onto a timeline. SituatedQA's temporal split and TimeQA provide, for each fact, a sequence of values with validity intervals. A free-text answer must be matched to one of those values, or to none. Structure alone would reject aliases and spelling variants. Distributional similarity alone would merge a predecessor with a successor who co-occurs with the same entity. This is the division of labor A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches describes: distributional methods give recall, curated structure gives precision. We therefore generate candidate matches with embedding similarity and accept a match only if it agrees with the timeline record. Each answer $a_k$ receives an interval $I(a_k) = [s_k, e_k]$ during which it was true.
The third part is the staleness score. Let $t_q$ be the query date. Let $K_{\text{now}}$ be the set of "currently" and "most recently" frames, and let $\mathbb{1}[\cdot]$ equal 1 when its condition holds and 0 otherwise. We define
$$\text{Stale}(q) = \frac{1}{|K_{\text{now}}|}\sum_{k \in K_{\text{now}}} \mathbb{1}\left[e_k < t_q\right],$$
the fraction of present-tense frames whose answer maps to an interval that has already ended. We also record an implied knowledge date: the latest interval end among the answers the model gives under the past-year frames. When this date comes out consistent across many facts for one model, it doubles as a sanity check that the probe recovers something like the model's training cutoff.
When $\text{Stale}(q)$ exceeds a threshold, the system retrieves. It then puts the timeline's current value, or the retrieved passage, into the prompt as the answer to override, rather than as optional context. This responds to RealTime QA's observation that models fall back on memory when retrieved evidence is weak.
Reproducibility is built in as a constraint, not a separate contribution. True Few-Shot Learning with Language Models shows that choosing prompts on a large held-out set inflates few-shot results. In a truly few-shot setting, selection barely beats chance, with about a 27% chance of picking a prompt 13% worse than average. Accordingly, the time-frame templates and the threshold are fixed on a small development slice and frozen before any test run. All runs use open-weight models with documented training cutoffs. We release the perturbation sets and per-instance answer–interval mappings. This also follows A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation, which identifies undocumented prompt conventions as a main source of irreproducible evaluation. We deliberately do not use clinical records. MIMIC-III's date-shifting, described in MIMIC-III, a freely accessible critical care database, removes exactly the real calendar anchors this probe depends on. We state this as a scope limit rather than a gap we fill.
Baselines
The naive reference points are closed-book answering, which never retrieves, and always retrieving. On RealTime QA, GPT-3 scored 44.9% closed-book and 66.5% with Google search on multiple-choice RealTime QA. Our open-weight models will not reproduce those absolute numbers, so we rerun both settings ourselves with an identical retriever. A useful selective method must approach the always-retrieve accuracy while retrieving on far fewer queries. At equal retrieval rate, it must beat random selection of which queries to retrieve for.
The strongest existing method for this exact decision is fact-duration adaptive retrieval from Mitigating Temporal Misalignment by Discarding Outdated Facts. It uses a BERT cloze model to predict how long a fact stays valid and retrieves when the gap between query date and evidence date exceeds that lifespan. On SituatedQA it reports an Exact Match of 20.5 while retrieving for 45% of queries, along with a 50–60% reduction in expected calibration error. We rerun it with the released setup on the same SituatedQA split, the same generator and the same retriever. Beating it means higher Exact Match at a 45% retrieval rate, and a dominating curve of accuracy against retrieval rate across thresholds. The duration model costs one small forward pass, while our probe costs $K$ generator passes, so we also report accuracy per unit of compute.
The strongest sensitivity-based competitor is SENSEL, using instruction paraphrases and demonstration reordering with the same $K$ and the same generator. On GPT-J-6B, On the Relation between Sensitivity and Accuracy in In-Context Learning reports +4.1 average points of area under the F1–coverage curve over maximum softmax probability. We found no reported result on Lacuna for SENSEL on SituatedQA, RealTime QA or TimeQA. We also run maximum softmax probability and entropy as confidence baselines.
Two ablations isolate the key ingredient. The first keeps the time-frame prompts but replaces timeline mapping with raw disagreement among answers. This tests whether the structured record matters or whether time-anchored paraphrases alone suffice. The second keeps timeline mapping but applies it to surface paraphrases. This tests whether the time frames matter.
Why not just…?
Why not just use SENSEL's ordinary paraphrase disagreement? It is the natural competitor and has published gains. However, the evidence that stale errors are consistent rather than random is direct. Rethinking Hallucinations finds prompt-agnostic errors dominate some benchmarks, and RealTime QA documents the outdated trap. Our first experiment decides this: if surface disagreement separates changed facts from stable ones as well as the staleness score does, the proposal collapses to SENSEL on temporal data, and we would report that.
Why not just always retrieve? RealTime QA attributes 34% of errors to retrieval failures, versus 13% to reading comprehension. Rethinking Hallucinations shows retrieval itself adds sensitivity to phrasing. SituatedQA finds that even swapping in an updated corpus leaves models 15 points worse on changed facts. Always retrieving pays those costs on every query, including the roughly 84% of information-seeking questions whose answers SituatedQA finds do not change. Selective retrieval is only worthwhile if it matches always-retrieve accuracy, and the accuracy–retrieval-rate curve measures that directly.
Why not just prepend today's date? RealTime QA reports that date prepending helps open-book models but hurts closed-book ones. We include it as a baseline in both settings.
Why not steer activations toward the present, as TARDIS does? TARDIS shifts hidden states along a direction from old-period to new-period text and gains up to 19.2% on temporal classification. It has not been tested on open-ended factual QA, and a single direction cannot supply a specific new value, such as the name of a new officeholder. It is complementary rather than a replacement. We test it as an optional add-on for flagged queries, not as a competitor to detection.
Proposed Experiments
The first experiment is a measurement study. On the SituatedQA temporal split and RealTime QA's "past" (2,886 questions) and "real-time" (1,470 questions) sets, we compute surface sensitivity and $\text{Stale}(q)$ for each question. We do this across open-weight models with known cutoffs, including GPT-J-6B and Llama 3-70B-Instruct. We then plot both as functions of how long before or after the cutoff the fact changed. The decisive result is a dissociation. Surface sensitivity should stay roughly flat, or even fall, for facts that changed after the cutoff. The staleness score should rise sharply. The implied knowledge date should cluster near each model's documented cutoff.
The second experiment is selective retrieval. On SituatedQA, we compare Exact Match against retrieval rate for all baselines and ablations. The headline comparison is at the duration model's 45% operating point. On TimeQA we test whether timeline mapping generalizes to facts with dense histories.
The third experiment is a robustness check on the probe itself. We rerun it with a second, independently written set of time-frame templates, frozen in advance. If the ranking of methods changes, the probe has inherited the brittleness it is meant to measure, and we report that as a negative result.
References
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- RealTime QA: What's the Answer Right Now?
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches
- True Few-Shot Learning with Language Models
- A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
- MIMIC-III, a freely accessible critical care database
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- TARDIS: Mitigating Temporal Misalignment via Representation Steering