Dating Benchmark Items by Content, Not Publication Date

Live benchmarks keep evaluation honest by testing models on material published after their training ended. The rule rests on an assumption: that a recent document carries recent knowledge. It often does not. A 2025 case report can restate a diagnostic rule that textbooks have printed for decades. An LLM-reworded coding problem can keep the exact logic of a problem the model memorized. A "new" question can also carry an answer that was true before the cutoff and has since been overturned.

We propose to date each benchmark item by what it says rather than when it was published. The method breaks the item into the small factual claims that determine its answer and checks whether text written before the cutoff already contains those claims. The check must catch small shifts in meaning, such as "discussed" versus "performed", or "suggested" versus "established". Each item then falls into one of three groups: already known before the cutoff, genuinely new, or known but since overturned. Human experts review only the claims the automatic checker cannot settle.

The payoff is threefold. The proposal offers a concrete explanation for a live contradiction in the contamination literature. It turns obsolete knowledge from a nuisance into something a benchmark can measure. It also attacks the largest running cost of live expert benchmarks, which is paying specialists to read every item.

Core Claim

The contradiction is this. LiveClin samples recent PubMed Central case reports twice a year and has 239 physicians audit them. It reads GPT-5's roughly 10 percentage-point drop on post-cutoff cases as evidence of leakage-free evaluation. Test of Time shows the same kind of drop can vanish:

  • LLM-generated reasoning questions on post-cutoff arXiv papers show no decay, and even a +2.19 pp rise on physics.
  • Cloze questions on the same papers drop 3.83 pp for GPT-4o-mini.
  • Renaming variables in LiveCodeBench erases its temporal decay entirely.
  • Influence-function attribution, which traces an answer back to the training documents that shaped it, finds the source document 77.5% of the time on cloze items and 17.5% on transformed items.

Our claim is that both findings follow from one missing variable: propositional novelty, meaning whether the claims that decide an item's answer were already stated in pre-cutoff text. Publication date is a noisy proxy for it. Rewording raises surface novelty without raising propositional novelty.

If this is right, three things should hold. Model accuracy should track propositional novelty more closely than it tracks publication date. Transformed items that show no decay should score as low-novelty. And the subset of "known but overturned" items should expose stale parametric memory, which the file treats as a separate problem. RealTime QA documents that memory: when retrieval fails, models fall back on outdated training knowledge instead of abstaining.

We define an item's novelty score as

$$N(x) = \frac{1}{|P(x)|}\sum_{p \in P(x)} \big[1 - \max_{d \in D_{<t}} \mathrm{ent}(d, p)\big],$$

where the terms are:

  • $P(x)$ is the set of answer-determining claims extracted from item $x$.
  • $D_{<t}$ is a corpus of documents dated before cutoff $t$.
  • $\mathrm{ent}(d, p)$ is the checker's probability that document $d$ states claim $p$ with the same modality, quantifiers and numbers.

The superseded flag is separate from $N(x)$. It fires when a claim is entailed by pre-cutoff text and contradicted by later text.

Research Program

The first component is claim extraction restricted to what decides the answer. Earlier work decomposes whole summaries into atomic facts: SCRPO does this and then checks each fact with natural language inference (NLI), a classifier that decides whether one text implies another. uMedSum does the same with threshold-based segmentation. We reuse this decomposition machinery. The difference is that we keep only claims whose negation would change the benchmark's correct answer. We identify those claims by editing each one and re-scoring the answer key.

The second component is the drift-sensitive matching step. Plain NLI and embedding search are the obvious tools, but they miss the failure that matters here: near-identical wording with different meaning. We therefore add a structural check. FactGraph parses text into Abstract Meaning Representation, a graph of concepts and their semantic roles, and scores it with small graph adapters. It beat FactCC and QAGS by up to 15% balanced accuracy on role flips and modality errors. FactGraph only scales to sentence-level comparisons, which is a real limitation for documents. Our claims are sentence-sized, so that constraint fits this use rather than blocking it.

Two known weaknesses constrain the design:

  • Atomic decomposition misses shifts in tone and hedging (SCRPO).
  • Small numeric edits are hard to detect (Saama NLI4CT).

We handle both by routing, not by pretending to solve them. Claims with hedging markers or numbers near the checker's decision boundary go to humans.

The third component is expert triage. Human effort in LiveClin cost over 1,700 person-hours and $42,000 per refresh. Two findings suggest that effort can be focused:

  • uMedSum showed that filtering drafts automatically before surgeons see them improves what they receive.
  • Feedback Copilot showed that traffic-light confidence indicators make expert review more effective.

The same paper also warns that heavily flawed drafts overload reviewers. We therefore show experts single claims, each paired with its closest pre-cutoff passage, and never whole items. Their only question is "is this the same claim?" Their answers become labels that retrain the checker for the next refresh.

Baselines

The naive reference point is publication-date dating, which is what LiveClin and LiveBench use. Every item after the cutoff counts as novel. We run it on exactly the same items. The reported reference effects are LiveClin's roughly 10 pp GPT-5 post-cutoff drop and Test of Time's 3.83 pp cloze drop against no decay on transformed items. Beating this baseline means our novelty score explains significantly more variance in per-item model correctness than the date does, and remains significant when both enter the same regression.

The strongest existing method for deciding whether an item's content was in training data is influence-function attribution. We use EK-FAC influence functions through Kronfluence on OLMo2-7B-Instruct, following Test of Time, which reports a 77.5% source-document hit rate on cloze items and 17.5% on transformed items. It needs white-box access and the training corpus, and it is expensive. On OLMo2 we therefore treat it partly as a reference signal, alongside direct corpus search that a human checks. The comparison is fair because both methods see the same items and the same OLMo2 corpus. Our method wins if it matches or exceeds influence attribution at flagging transformed items whose logic appears in the corpus, where influence falls to 17.5%. It must also use only black-box text search, so it applies to closed models.

The key ablation replaces the drift-sensitive matcher with dense-embedding nearest neighbours plus plain NLI, holding the same retrieval budget and the same threshold-tuning split. The embedding route is weak on meaning-flipping edits: on ProCon.org NPOV data, overlap-based detectors reached about 84–85% ROC AUC against 95.3% for trained classifiers (Detecting Hallucination and Coverage Errors). A second ablation removes the answer-determining filter and scores all claims, to test whether that filter is what makes the score predictive.

For triage, the baseline is full expert review of every item, as in LiveClin. Success means at least 90% agreement with full-review verdicts at under a third of the reviewer time. No reported result was found on Lacuna for selective expert verification of live benchmark items.

Why not just…?

Why not just keep dating by publication and reword more aggressively? Aggressive rewording is exactly what launders contamination. Test of Time shows it removes the decay signal while influence attribution falls from 77.5% to 17.5%, meaning the model keeps its benefit and the audit loses its evidence. Experiment 2 tests directly whether date or novelty better predicts correctness.

Why not just run n-gram or embedding contamination search against the corpus? This is the serious threat. If embedding search plus ordinary NLI matches our checker, the structural layer adds nothing and we should drop it. Evidence against it is indirect. FactGraph and the NPOV study both show overlap and embedding methods missing role and modality flips, but neither tested contamination. The main ablation settles the question on our data. We report the fraction of items where the two methods disagree and let expert labels adjudicate.

Why not just ask a strong LLM whether each item is novel? LiveBench found that models which cannot solve a hard problem also cannot judge solutions to it. A judge model is also itself contaminated, so its sense of "familiar" is the quantity under test. We include an LLM-judge variant with the same retrieved passages as an extra baseline. We expect it to over-report novelty on transformed items.

Why not just pay experts to review everything? This works, but LiveClin's cost scales with every refresh. Experts also judge claim-level matches better when shown one claim than when reading a whole case. We measure this in Experiment 4 rather than assuming it.

Proposed Experiments

Experiment 1: calibrating the novelty score on a model with known training data. This uses OLMo2-7B, whose pretraining corpus is public. The items are RealMath/arXiv items in cloze and transformed form (Test of Time), plus original and perturbed LiveCodeBench problems. For a stratified sample of 400 items, we label corpus presence by exhaustive search followed by human confirmation. We report AUROC for the novelty score, the embedding ablation, and influence attribution.

Experiment 2: the decisive test of the core claim. It covers GPT-4o-mini and the other models in Test of Time, on the same arXiv and LiveCodeBench items plus LiveClin cases. The pre-cutoff corpus is public PubMed Central and arXiv text before each model's cutoff. We fit per-item correctness against publication date and novelty jointly. The claim fails if novelty adds no explanatory power beyond date. It also fails if transformed physics items do not score as low-novelty, since that low novelty is our proposed explanation for their +2.19 pp rise.

Experiment 3: testing the superseded group. This uses SituatedQA and RealTime QA, where answers change over time. For items flagged as superseded, we measure how often closed-book and search-augmented models give the old answer. The reference points are RealTime QA's 44.9% closed-book and 66.5% search-augmented multiple-choice accuracy. The prediction is that superseded items account for a disproportionate share of the 34% of errors that RealTime QA attributes to retrieval failure.

Experiment 4: human triage on a LiveClin-scale slice. We take 150 LiveClin cases. Four physicians review them in two arms: full-item review, and claim-level triage restricted to uncertain claims. We measure person-hours and agreement with the full-review verdict. We follow MI-CLAIM-GEN reporting of annotator qualifications and inter-annotator agreement.

If Experiment 2 fails, publication-date benchmarks are vindicated and we report that. If Experiment 1 shows the embedding ablation performs as well, the proposal reduces to a cheaper and simpler contamination audit, which is still worth having.