Language models are now widely used as automatic graders: a large model reads another model's answer and scores it, often with no reference answer to compare against. Every such grader has a training cutoff, the date after which its weights know nothing. When the world changes after that date, a grader asked whether "X is the CEO of Y" is correct will compare the answer to its own outdated memory. It will reward the stale answer and mark the correct, updated one as a hallucination. We call this temporal narcissism: a grader preferring answers that match its own frozen snapshot of the world. Nobody has measured how large this effect is. It could quietly invert leaderboards for any system that actually tracks current facts.
This proposal does two things. First, it measures the effect cleanly, using questions whose correct answer changed on a known date and graders with different known cutoffs. Second, it tests one specific fix, which we call a decoupled grader. The grader's job is split in two. A small language model only extracts the factual claims in an answer, and it is forced by a grammar to write them as well-formed records (subject, relation, value, asserted time). A deterministic program then checks those records against a timestamped fact store, and a learned model of how long facts stay true decides when the store itself is too old to trust. The bet is that the grader's memory has to be removed from the comparison step, not merely supplemented. Handing the grader fresh evidence in its prompt should not be enough, because models given current context still fall back on what they memorized.
Core Claim
Reference-free LLM graders systematically penalize correct post-cutoff facts. We expect this penalty to switch on sharply at each grader's own training cutoff. Prompting the grader with retrieved, dated evidence will reduce the penalty but not remove it. A grader whose factual comparison never passes through its own weights will remove it, and will cost little at evaluation time.
We define the effect as a single quantity. For a grader $J$ and a set $Q$ of questions whose answer changed at time $t_c$, let $s_J(a)$ be the score $J$ gives answer $a$. The temporal penalty is
$$\Delta_J(t_c) = \mathbb{E}_{q \in Q}\left[ s_J(a^{\text{current}}_q) - s_J(a^{\text{stale}}_q) \right].$$
Here $a^{\text{current}}$ is the answer true at query time and $a^{\text{stale}}$ is the answer true before $t_c$. A sound grader has $\Delta_J > 0$ regardless of $t_c$. The prediction is that a monolithic grader has $\Delta_J > 0$ only when $t_c$ precedes $J$'s cutoff, and $\Delta_J \le 0$ after it.
The motivation comes from three findings that have not been put together. LLMs as Narcissistic Evaluators shows that graders inflate scores for text from their own model family, most strongly when no reference is given. It studies family style, not time. SituatedQA shows that 16.5% of information-seeking questions change answers over time. It also shows that models fed an updated corpus are still 15 points less accurate on updated facts than on stable ones. Can Agent Memory Systems Track Evolving State? shows that this "state drift" persists even when the correct, updated evidence is guaranteed to be in context, and that chain-of-thought often makes it worse by rationalizing the outdated fact.
The third finding is the one that shapes our design. If oracle context cannot override memory for a generator, there is little reason to believe it will for a grader. This is why we take the comparison step out of the model's hands entirely.
Research Program
Benchmark. The first piece is a temporal grading benchmark built from the temporal split of SituatedQA, which pairs questions with answers valid over dated intervals. For each question we produce three candidate answers in matched surface form: the current answer, the most recent superseded answer, and a plausible wrong answer.
We then stratify questions by change date relative to the cutoffs of several open-weight grader families. Open weights are needed because grammar-constrained decoding requires access to the model's next-token scores, and we want the same families in every condition. Because ground truth is known by construction, no human labels are needed for the core measurement.
We add a second, agentic slice from StateMemBench. Its multi-session scenarios have an explicitly superseded value and a current value, which tests whether the effect carries over from world facts to in-session state. This slice is the harder setting, because the "store" must be built from the conversation itself.
Decoupled grader. The second piece is the grader itself, assembled from three existing parts.
Claim extraction reuses the grammar-constrained decoding of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. That paper masks out any token that would break a context-free grammar. It reports 100% structural validity on parsing, against about 64% unconstrained, at 1–4 ms per token. Its input-dependent grammars restrict entity choices to spans present in the input, and we use them so that extracted subjects must come from the candidate answer.
The paper also states plainly that the grammar does not make content correct. That is exactly why we never ask the grammar-constrained model to judge truth: it only transcribes claims. This also closes a weakness the StateMem paper leaves open. StateMem generates its structured state units without constraints, so a single malformed record can break its deterministic checker.
The check itself is a lookup into a timestamped fact store, built offline. This mirrors the offline-precompute, online-lookup split used in recommenders such as DisCo and Discrete Semantic Tokenization for Deep CTR Prediction. Those systems move heavy language-model work offline so that serving costs milliseconds. We borrow the split but use it for evaluation, and unlike them we must handle the store going stale.
For that we reuse the fact-duration model of Mitigating Temporal Misalignment by Discarding Outdated Facts. That model predicts how long a fact typically stays true and discounts evidence older than that span. It predicts duration within three years for 65% of temporally dependent facts and cut calibration error by 50–60% on SituatedQA. Applied here, a claim whose stored evidence has outlived its predicted duration is scored "unverifiable", not "wrong". Penalizing it would reintroduce the same bias from the store side.
Claim-level verdicts are then combined with non-factual sub-scores, such as fluency, by a shallow aggregator in the style of HD-Eval. HD-Eval decomposes grading criteria into a tree and fits a small model on top. Its tree is static and its factual sub-criteria are still judged from model memory, which is precisely the part we replace.
Ranking analysis. The third piece asks whether temporal bias changes which system wins, not only how single answers are scored. Following Rethinking Evaluation Metrics for Grammatical Error Correction, we convert per-answer scores into pairwise matches and rate systems with TrueSkill, a skill-rating method originally built for ranking game players. That paper raised the correlation between the IMPARA metric and human rankings from 0.887 to 0.975 this way. We will report rankings under both plain averaging and tournament aggregation.
Baselines
The naive reference point is a monolithic reference-free grader. A single open-weight model of each family is prompted to rate factual correctness on a fixed scale, with today's date stated in the prompt. This is how most reference-free grading is done in practice. Its $\Delta_J$ curve across change dates is the quantity we expect to flip sign at the cutoff.
The strongest existing competitor is a retrieval-augmented grader. The same model receives the same timestamped evidence our store would return, formatted with dates, plus an explicit instruction that facts may have changed and dated evidence overrides memory. It is the strongest competitor because it uses identical information to the decoupled grader and differs only in where the comparison happens. If it matches us, the decoupling is unnecessary.
We will run it fairly. Retrieval output is identical in both conditions, including the oracle setting where the correct dated fact is always present. Prompt wording receives the same tuning budget as our aggregator, about 5% of labeled pairs, the amount HD-Eval found sufficient. Each family is used as its own grader in both conditions, so model size is held fixed.
No reported result was found on Lacuna for any LLM grader, retrieval-augmented or not, on temporally updated facts. The closest reported numbers measure different targets and serve only as context. HD-Eval reports about a 15% Pearson gain over plain GPT-4 prompting on SummEval. The duration-aware system reports 20.5 exact match on SituatedQA while retrieving for 45% of queries. SituatedQA's best retrieval QA model reaches 23% exact match against about 57% for humans.
Beating the retrieval-augmented grader means two things. On questions changed after the grader's cutoff, our $\Delta_J$ must be positive and statistically indistinguishable from its value on pre-cutoff changes, while the retrieval-augmented grader keeps a significant post-cutoff drop. And this must hold without losing accuracy on the stable-fact control set.
A third, older baseline is reference-based exact match against SituatedQA's current gold answer. It is an upper bound wherever a current reference exists, and it shows how much of the gap a fresh reference alone would close.
Three ablations isolate our ingredients. The key one is the evidence-in-prompt ablation: the retrieval-augmented grader above is literally the decoupled grader with the deterministic comparison replaced by the model reading the evidence. Removing the grammar constraint from extraction measures how often malformed or invented claim records corrupt verdicts. Removing the duration model measures how much the store's own staleness creates false penalties.
Why not just…?
Why not just give the grader retrieved, dated evidence in its prompt? This is the real threat, and it is the main comparison above, not a strawman. The case against it is indirect but strong. StateMem found state drift persisting under oracle retrieval, and SituatedQA found a 15-point gap on updated facts even with an updated corpus. Both are generation results, though, and graders may behave differently. If the retrieval-augmented grader's post-cutoff penalty disappears in our oracle setting, we will report that the problem is solved by prompting. The contribution then narrows to the measurement and the benchmark.
Why not just use reference-based grading with a fresh gold answer? Where a current reference exists, it should work, and we include it as an upper bound. But reference-free grading is used precisely where references do not exist, such as open-ended answers and agent outputs. References written at benchmark-creation time also become stale answers themselves as time passes, so this approach moves the obsolescence problem rather than removing it.
Why not just state the current date and tell the grader to be uncertain about recent facts? This is folded into every baseline prompt, so its effect is measured directly. The worry is that it trades one error for another: a grader that hedges on everything recent will stop penalizing stale answers too. That would show up as a shrinking but still non-positive $\Delta_J$, and as lost accuracy on the wrong-answer distractors.
Why not just decompose criteria as HD-Eval does? Decomposition fixes how criteria are weighted, but each factual sub-criterion is still answered from the grader's memory. We include HD-Eval-style decomposition with memory-based factual sub-scores as a variant. If decomposition alone removes the penalty, our store and duration model add nothing.
Proposed Experiments and Decisive Checks
The primary experiment computes $\Delta_J(t_c)$ for every grader family and condition, binned by change date relative to each grader's cutoff. Since several families have different cutoffs, the same question acts as "pre-cutoff" for one grader and "post-cutoff" for another. This is a natural experiment that controls for question difficulty.
The decisive pattern is a step down in $\Delta_J$ at each monolithic grader's own cutoff, and a flat curve for the decoupled grader. If no step appears for the monolithic graders, the temporal-narcissism hypothesis is wrong and we will say so.
A cross-family check pairs temporal staleness with the family self-preference documented in the Narcissus paper. It asks whether a grader is especially forgiving of stale answers produced by its own family, which would compound the two biases.
A ranking check simulates systems with different refresh policies, from never-updated to updated daily. It asks whether averaged monolithic scores rank the never-updated system above the fresh one, and whether TrueSkill aggregation or the decoupled grader restores the correct order.
On the StateMemBench slice, we test whether the same pattern holds when the store is the conversation's own state records. Here we report grammar-constrained extraction validity against unconstrained extraction.
Finally, we report store coverage: the fraction of claims the store cannot resolve, scored "unverifiable". We also report per-answer latency of the decoupled grader against a single monolithic grader call. A low penalty bought with high unverifiable rates would not count as success, so coverage is reported alongside every $\Delta_J$.
References
- LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- Can Agent Memory Systems Track Evolving State?
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- DisCo: Towards Harmonious Disentanglement and Collaboration between Tabular and Semantic Space for Recommendation
- Discrete Semantic Tokenization for Deep CTR Prediction
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
- Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?