Language models are now routinely graded by other language models, a practice called "LLM-as-a-judge." In medicine and law, the facts being graded change. Treatment guidelines are revised and statutes are amended. A judge model trained before a revision will mark a correct, up-to-date answer as a hallucination, and it will do so with confidence, because it is sampling the same outdated fact from its frozen weights every time. We propose a simple test that exposes these stale verdicts. The judge grades each atomic claim in an answer twice, once against the previous version of the governing document and once against the current version. A claim whose verdict flips between the two versions is exactly a claim where obsolescence matters. Such claims get their own statistically calibrated acceptance threshold, and when the threshold is not met they go to a human expert. Claims whose verdict is stable can be graded automatically with little risk.
The opportunity is that two literatures have each solved half of this. Risk-controlled judging gives provable error bounds, but only for binary trivia questions. Work on knowledge obsolescence measures how stale benchmarks mislead evaluation, but only in general-domain QA and only for stale answer keys, not stale judges. Neither line asks whether the evaluator itself is out of date, and neither works in professional domains, where an outdated verdict can endorse a superseded drug regimen.
Core Claim
Our central claim is that the most informative signal for routing a domain-specific evaluation to a human is version disagreement. A claim is version-sensitive if the judge's verdict changes when it is shown the old rather than the new authoritative source. This signal is distinct from ordinary judge uncertainty, and is often uncorrelated with it. We predict that on version-sensitive claims, standard uncertainty signals look reassuring while error is high. Sampling variance and predictive entropy are both low because the judge is consistently wrong, and retrieving only the current document still leaves residual errors, because the judge's prior or a stale passage can override the evidence it was given.
If the prediction holds, then three things follow. First, splitting claims by version sensitivity and calibrating a separate acceptance threshold for each group should keep the false-acceptance rate below a target on both groups. Pooled calibration should fail this test on the version-sensitive group. Second, the split should let the system accept far more stable claims automatically. Third, the human budget should concentrate where obsolescence actually bites.
To state the signal precisely, let $c$ be an atomic claim and $J(c \mid E)$ be the judge's verdict given evidence $E$. Let $E_{\text{old}}$ and $E_{\text{new}}$ be the superseded and current versions of the governing source. The version-flip indicator is
$$f(c) = \mathbb{1}\left[J(c \mid E_{\text{old}}) \neq J(c \mid E_{\text{new}})\right].$$
Within each group $s \in \{0,1\}$ defined by $f$, we pick an acceptance threshold on a black-box confidence score $q(c)$. This score is the agreement rate over $k$ sampled verdicts under $E_{\text{new}}$, so the method needs no logit access. The threshold is chosen so that the Clopper–Pearson upper confidence bound on the false discovery rate (the fraction of accepted verdicts that are wrong) among accepted calibration claims is at most a user-chosen $\alpha$. Claims below threshold go to experts.
Where This Sits
The method we reuse comes from Judge, Retrieve, or Abstain. That paper routes a judge between parametric judgment, judgment with web retrieval, and abstention to humans, and it proves false-discovery-rate control with Clopper–Pearson bounds. It is limited to binary correctness on TriviaQA, NQ-Open, HotpotQA, and PopQA. It requires logits, and it assumes retrieval returns current truth. We keep the risk-control machinery and change three things. We grade atomic domain claims instead of whole trivia answers. We add version contrast as a stratifying signal. We replace entropy with sampled agreement, which also works with closed APIs.
The atomic decomposition comes from LeMAJ. It splits legal answers into "Legal Data Points," tags each one Correct, Incorrect, Irrelevant, or Missing, and reaches a Pearson correlation of 0.700 with lawyers on LegalBench. Its triage of answers for human review is heuristic, and it treats source documents as fixed. We inherit its claim granularity and replace its heuristic triage with calibrated, version-aware routing.
Three papers motivate the gap. When Benchmarks Age shows that stale answer keys mislead evaluation, with an Evaluation Misleading Rate above 20% for GPT-4o-mini. It also documents a "Context Trap," in which models prefer outdated retrieved text over their own correct knowledge. Its fix is to refresh the answer keys, which does nothing for a reference-free judge whose knowledge is stale. SituatedQA reports that models given an updated Wikipedia corpus were 15 percentage points less accurate on updated facts than on stable ones. That is direct evidence that handing a model the new document is not enough. The CHB guideline study shows that GPT-4's guideline coherence on Chronic Hepatitis B vignettes rose from 51% to 91% when the 2024 WHO guideline was attached. This result cuts both ways for us. It shows that stale knowledge is a real clinical problem. It also shows that context helps a lot, which is the main threat to our proposal and is addressed below.
Mitigating Temporal Misalignment predicts how long a fact will stay true and discounts confidence when evidence is old, cutting calibration error by 50–60% on SituatedQA. It works on QA generation rather than evaluation, and it predicts lifespans without knowing start dates. Version contrast avoids the lifespan-prediction problem: when a revision history exists, we observe the change directly instead of predicting it.
Research Program
The first component is a testbed. No versioned, claim-level, expert-labeled judging benchmark appears in the literature we reviewed, so we build one in two domains where revision histories are public and dated.
In medicine, we start from the 2015 and 2024 WHO Chronic Hepatitis B guidelines used in the CHB study. With clinician collaborators, we add further guideline pairs that have documented revisions. For each pair, clinicians write vignettes whose correct management differs between versions, plus control vignettes whose answer did not change.
In law, we pair statute or regulation texts before and after an amendment. We follow LegalBench's task styles of rule-recall and rule-application, so the items are recognizable to legal annotators.
For each vignette, we generate candidate answers from several models. Some are prompted to follow the old version, some the new, and some are left unprompted. This gives us a mix of current-correct, stale, and plain-wrong claims. Answers are decomposed into atomic claims using LeMAJ-style prompting. Domain experts label each claim as current-correct, stale (correct under the old version only), incorrect, or irrelevant. A held-out portion of these labels serves as the calibration set.
The second component is measurement. On this testbed we quantify the stale-judge problem, the reverse of the "Correct but Wrong" paradox, using two rates. The obsolescence false-rejection rate is how often a judge marks a current-correct claim as wrong. The stale-acceptance rate is how often it accepts an outdated claim. We measure both for parametric judging, for judging with the current document in context, and for judging with retrieval over a mixed-version corpus. To our knowledge this is the first measurement of judge-side obsolescence in a professional domain.
The third component is the version-contrast router and its calibrated guarantee. Beyond the binary verdict, we study whether risk control extends to LeMAJ's four-way tags. We do this by controlling the false discovery rate separately for "accepted as Correct" and "accepted as Incorrect" decisions. Missing-information detection, which LeMAJ reports as hard, is left to experts by design.
Baselines
The naive reference point is a single frontier judge prompted with a rubric, using only its parametric knowledge, with no human routing. It shows how much damage stale knowledge does when nothing is guarded. As a fixed-reference baseline, we score answers with ROUGE-L against expert reference answers. LeMAJ reports a correctness correlation of 0.170 for ROUGE-L on LegalBench, and we expect it to be blind to version changes as well.
The strongest existing method is Judge, Retrieve, or Abstain, adapted to our testbed. It uses the same judge models and the same calibration split. Retrieval runs over a corpus containing both document versions, which mirrors real search results that mix old and new material. Where logits are available, we run it with entropy as published, and we also run it with sampled agreement so the uncertainty signal matches ours. At $\alpha = 0.20$, that paper reports that combined parametric and retrieval judging raised coverage on NQ-Open from 7% to 82% with Qwen3-8B, and on HotpotQA from 40% to 85% with LLaMA-3.1-8B. We found no reported result on Lacuna for this method, or any risk-controlled judge, on clinical or legal claims that change over time.
A second strong competitor is the "oracle current document" judge. It is given exactly the current guideline in context, mirroring the CHB contextualized condition, and is wrapped in the same pooled conformal calibration. This baseline is the most dangerous one for our claim.
We also run SURE's self-consistency routing, from Towards Reliable LLM Grading, which flags high score variance across 20 perturbed prompts. That paper reports a 40–90% reduction in grading time in Python coursework. Finally, we run LeMAJ's heuristic triage, which clears high-scoring answers.
Three ablations isolate the key ingredient. The first stratifies by a random flip label with the same group sizes. The second computes version flips at the level of whole answers rather than claims. The third removes stratification entirely, which is equivalent to the oracle-document baseline.
For every method, the judge models, sample budget, calibration split, and threshold search grid are identical, and none is tuned on test data. We count our method as beating the baselines if it meets two conditions at the same $\alpha$. First, it must keep the empirical false discovery rate at or below $\alpha$ on version-sensitive claims in cases where the pooled baselines exceed it. Second, its overall coverage must be at least as high, with a smaller share of expert time spent on stable claims.
Why not just…?
Why not just put the current guideline in the judge's context? This is a real threat. The CHB study's jump from 51% to 91% coherence suggests that context may fix most stale verdicts. However, SituatedQA's 15-point deficit on updated facts and the Context Trap in When Benchmarks Age both suggest that residual errors concentrate on the claims that changed. Our experiment compares the oracle-document baseline against version contrast. If the oracle baseline already holds the false discovery rate below $\alpha$ on version-sensitive claims without stratification, then version contrast adds nothing for obtaining the guarantee. In that case we will report that the remedy is simply "attach the right document," and the measurement study becomes the contribution. Version contrast would still matter in the realistic setting where the judge cannot tell which retrieved document is current, so we also test the mixed-version corpus condition.
Why not just use self-consistency, as in SURE? Consistency measures agreement, not currency. A judge whose knowledge predates a revision will reproduce the old fact on every sample. SURE itself notes that majority voting "didn't eliminate bias." We test this directly by plotting error against sampling variance, separately for version-sensitive and stable claims. Our prediction is that on version-sensitive claims, low variance coexists with high error. If that prediction fails, SURE-style routing is sufficient.
Why not just refresh the reference answers, as in When Benchmarks Age? That pipeline reached 89.52% accuracy in recovering current answers for general questions. It repairs answer keys, but professional judging is usually reference-free, as in LeMAJ, and the judge's own knowledge goes stale whether or not keys exist. Refreshing references also gives no error guarantee on accepted verdicts. We include a reference-refreshed judge in the oracle-document comparison, so this alternative is covered empirically.
Why not just fine-tune a domain classifier as the judge? A classifier is frozen at its training labels and must be relabeled after every revision. PubMedQA's fine-tuned BioBERT reached 68.1% against 78.0% for experts, even before any revision. We do not build this baseline at scale. We consider it dominated on cost of maintenance, not necessarily on accuracy, and we say so.
Proposed Experiments
The decisive check is a stratified risk audit. We run every method at $\alpha \in \{0.05, 0.10, 0.20\}$ with repeated random calibration/test splits. For each method we report the empirical false discovery rate separately on version-sensitive and stable claims, along with overall coverage and the expert-hours implied by abstentions. The proposal succeeds if pooled methods violate $\alpha$ on version-sensitive claims while stratified version contrast does not, at comparable or higher coverage. The proposal fails if the oracle-document baseline already satisfies $\alpha$ on both groups.
The second check measures how much the flip signal costs and what it misses. A claim can be stale even when no superseded document is available to contrast against. To estimate this, we withhold $E_{\text{old}}$ for a subset of revisions and check how many stale claims go undetected.
A third check is a general-domain sanity run. We use the time-sensitive items that When Benchmarks Age identifies in BoolQ and NQ, with their old and updated answers standing in for the two versions. This lets us compare against Judge, Retrieve, or Abstain on ground close to its published setting.
Finally, we report whether claim-level decomposition, the ingredient taken from LeMAJ, is necessary. If flips computed on whole answers route just as well, the simpler design wins.
References
- Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
- LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
- When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- The Impact of Access to Clinical Guidelines on LLM‐Based Treatment Recommendations for Chronic Hepatitis B
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Towards Reliable LLM Grading Through Self-Consistency and Selective Human Review: Higher Accuracy, Less Work