Researchers increasingly use a large language model as a grader, an "LLM judge" that reads another model's open-ended answer and decides whether it is correct or good. Its trustworthiness is usually established by showing it agrees with humans on existing benchmarks. Those benchmarks, however, are mostly old, mostly about Western topics, and largely sitting in the judge's own training data. This proposal tests a specific worry. A judge may grade well mainly when it already knows the answer. When it grades answers about events after its training cutoff, or about places its training data barely covers, it may fall back on whether the answer sounds right. If so, published judge–human agreement figures are themselves a product of contamination (test material leaking into training data), and they overstate how reliable the judge is on fresh or non-Western content.
The core idea is simple to state. Time and geography may break a judge in the same way, because both remove the judge's prior knowledge of the facts it is checking. We will measure this directly. A difference-in-differences design will grade the same items with judges whose cutoffs fall before and after each event, crossed with how well each item's region is covered in pretraining text. We will then test whether a cheap per-item probe of the judge's own knowledge can predict where its verdicts should not be trusted.
Core Claim
The claim is that LLM-judge error on factual open-ended grading is driven by a single variable, the judge's prior knowledge of the item's facts. Being past the judge's cutoff and coming from a thinly covered region are two routes to the same deficit. The deficit shows up as a specific failure: the judge accepts plausible fabrications and penalises correct but unfamiliar answers. It does not show up as random noise. Three predictions follow. A judge grading post-cutoff items from well-covered regions should behave like the same judge grading pre-cutoff items from poorly covered regions. The two deficits should compound, so the worst cell is post-cutoff items from poorly covered regions. And the judge's closed-book accuracy on an item's underlying facts should predict its grading error better than its own stated confidence.
This differs from the closest existing work in where it places the cutoff. The Silent Judge showed that judges shift verdicts by up to +30% (GPT-4o on ELI5) when identical content is merely labelled "NEW (2025)" rather than "OLD (1950)", and that they never admit using the label. That work studies a textual cue. We study a real chronological split relative to the judge's training. Preference Leakage treats judge contamination as overlap between the judge and the model that produced the training data, measured on generic conversational benchmarks. We treat it as overlap between the judge and the facts being graded, and stratify by region. LiveBench rejected LLM judges because of a "judging gap": models cannot grade problems beyond their own ability. We test a knowledge-based version of that gap that LiveBench did not separate out, which is whether the judge can grade what it does not know even when the reasoning involved is easy.
Why the pieces fit
Each direction has a clear role here. Chronological partitioning supplies the main experimental lever. Geographic generalization supplies the second axis and the stakes. Contamination is the explanation under test. The LLM judge is the object being audited.
Several papers motivate the geographic axis. Large Language Models are Geographically Biased found that models carry strong geospatial knowledge on aggregate (Spearman ρ = 0.83 for infant mortality with GPT-4 Turbo) yet systematically misestimate population density in India and Africa. That pattern suggests patchy rather than absent regional knowledge. Cultivar showed that 32 open-weight models translate US-localised text better than text localised to its native region, with named-entity errors causing over 80% of locale-specific failures. That makes entity substitution a natural way to build plausible fabrications. How Reliable is Multilingual LLM-as-a-Judge? found judge agreement around Fleiss' κ ≈ 0.3 across 25 languages. That study confounds language with region, which we avoid by keeping everything in English.
Test of Time constrains the design. It showed that a drop in performance after a cutoff disappears when questions are rewritten, and that influence-function tracing to the source document falls from 77.5% to 17.5% after semantic transformation. A naive "accuracy dropped after the cutoff" comparison is therefore unreliable. For this reason we hold items fixed and vary the judge's cutoff, rather than comparing different items before and after a date. Unveiling Causal Reasoning in LLMs provides a reusable recipe for building post-cutoff items from dated news, together with a membership-inference freshness check (Min-K% Prob). Its news, though, comes only from BBC and The Guardian.
Research Program
We will build a region-stratified, dated corpus of factual open-ended questions. Each question is derived from a news article about a local event. Articles come from outlets in roughly a dozen regions spanning high and low pretraining coverage, and their publication dates straddle the cutoffs of the judges we study. All items are in English, using English-language local outlets or professional translation, so that language is not confounded with region. CausalProbe-2024 items serve as a Western, already-validated anchor. Each question receives three kinds of answer, all with known correctness: a correct reference-quality answer written by annotators from the source article; a correct answer paraphrased in a regionally natural register; and a fabricated answer that is fluent and plausible. Fabrications are built by Cultivar-style entity substitution (swapping a local official, place, or unit for a plausible alternative), by shifting dates or numbers, or by inventing a causal claim. Because correctness is set by construction and checked by humans, the judge's verdicts can be scored against a known answer key.
We quantify regional coverage directly. For judges with open training data, such as OLMo2, the variable is document frequency of the item's key entities in the pretraining corpus. For other judges we use FineWeb (FineWeb) counts as a proxy.
The central analysis is a regression over judge $j$ and item $i$:
$$\text{err}_{ij} = \alpha_i + \beta_j + \gamma\,\text{post}_{ij} + \delta\,\text{lowcov}_i + \eta\,\text{post}_{ij}\cdot\text{lowcov}_i + \epsilon_{ij}$$
Here $\text{err}_{ij}$ is whether judge $j$ misgrades item $i$, $\alpha_i$ and $\beta_j$ absorb item difficulty and overall judge quality, $\text{post}_{ij}$ indicates that the event postdates judge $j$'s cutoff, and $\text{lowcov}_i$ indicates a low-coverage region. The item effects matter. Because the same item is post-cutoff for an older judge and pre-cutoff for a newer one, $\gamma$ isolates the judge's missing knowledge from item hardness. That addresses the confound Test of Time identified. The interaction $\eta$ tests compounding. Finally, we ask whether $\delta$ and $\gamma$ act through the same channel: if both shrink to near zero once we condition on the judge's closed-book accuracy on the item's facts, the "single knowledge deficit" claim holds.
The constructive part of the program is a per-item knowledge probe. Before grading, we ask the judge two to four short closed-book questions about the facts the item depends on, and score its answers against the source. The probe's accuracy becomes a trust score that decides whether a verdict is accepted or the item is routed to reference-guided grading or a human. We compare this probe with the judge's verbalized confidence and its log-probability on the verdict.
Judges will span families and cutoffs: GPT-4o and Gemini-2.5-Flash (the judges audited in The Silent Judge), plus Qwen2.5, Llama 3.1, and OLMo2 at several sizes. For OLMo2 the training data is known, so coverage can be measured exactly rather than estimated.
Baselines
The naive reference point is to assume the judge's reliability is uniform and equal to its published agreement on standard benchmarks. Concretely, we measure each judge's accuracy on a pre-cutoff, high-coverage slice and treat that figure as its reliability everywhere. The proposal beats this baseline if accuracy on the other three cells falls significantly below it, specifically if a fixed-effects estimate of $\gamma$, $\delta$, or $\eta$ excludes zero. For context, the best multilingual judge agreement reported is Fleiss' κ = 0.54 for GPT-4o on WikiLingua binary correctness, falling to 0.32 for 1–5 grades (How Reliable is Multilingual LLM-as-a-Judge?). We found no reported result on Lacuna for judge accuracy on post-cutoff, region-stratified factual grading.
The strongest existing competitor is reference-guided judging, in which the judge receives the source article or a gold answer alongside the response. The rubric-and-reference prompting described in Evaluating Generative AI with Large Language Models is the standard practice. It is the strongest competitor because it supplies exactly the knowledge whose absence we hypothesise is the problem. A close variant is retrieval-augmented judging, where the judge searches for evidence itself. For a fair comparison, every variant uses the same items, the same judge checkpoints, temperature 0, the same rubric text, and the same per-item token budget, with prompt wording tuned once on a held-out pre-cutoff slice. We found no reported result on Lacuna for reference-guided or retrieval-augmented judge accuracy on fresh regional news.
The probe "beats" this baseline in two senses. First, it must reach comparable accuracy over the items it accepts while sending only a minority of items (say under 30%) to reference-guided grading, which lowers cost. Second, and more importantly, retrieval-augmented judging may itself fail in low-coverage regions, because search results for thinly covered places are sparser. If so, the probe should catch failures that retrieval misses.
A third baseline, the distributional judge of Distributional LLM-as-a-Judge, outputs a calibrated distribution over scores. It reports KL divergence of 0.23 against 0.60 for single-point alignment on SNLI. We will use its spread as an alternative trust signal, to test whether general uncertainty modelling already captures what our knowledge probe captures.
The key ablation replaces the item-specific probe questions with generic questions about the same region. If the generic version works as well, then the signal is "this region is unfamiliar," not "this specific fact is unknown," and a simple region lookup table would suffice.
Why not just…?
Why not just use programmatic scoring, as LiveBench does? LiveBench itself concedes that it cannot grade open-ended answers, and factual answers about local events are written in free text that string matching handles poorly. We will still use LiveBench as a check. On its objective items we compare judge verdicts with the programmatic ground truth, split by whether the item postdates the judge's cutoff. This gives a version of our $\gamma$ estimate in which the answer key involves no human annotation. If $\gamma$ is near zero there but large on our news items, the effect is about factual knowledge rather than reasoning, which is our claim.
Why not just always give the judge a reference answer? This is the real threat, and we take it seriously. If reference-guided judging erases every cell difference, the main result becomes a usage rule, "never grade fresh or regional content without a reference," and the probe's value shrinks to cost savings. We think the gap may not fully close for two reasons. Entity-substitution fabrications require the judge to notice that a plausible local name does not match the reference, and Cultivar shows that entity handling is exactly where models are weakest outside US contexts. And in practice references are often unavailable for fresh content, which forces retrieval, whose coverage is itself regionally skewed. The reference-guided arm of the experiment settles the question either way, and we will report it as a primary result, not a footnote.
Why not just ask the judge to reason step by step and state its confidence? The Silent Judge found a 0% rate of judges acknowledging the cues that drove their verdicts. Their rationales were post-hoc stories. Preference Leakage found that judges could not identify their own students in blind tests, yet still favoured them. Both results suggest that self-report is a poor window into what drives a verdict. The direct comparison between verbalized confidence and the closed-book probe tests whether this carries over to knowledge gaps. If verbalized confidence predicts error as well as the probe does, we will say so, and the cheaper method wins.
Why not just rely on a strict temporal cutoff for the evaluated model, as CausalProbe-2024 does? That approach protects the evaluation from the student having memorised the answers. It does nothing about the judge lacking knowledge of the items. Test of Time further shows that date-based signals can be laundered by rewriting. Our design varies the judge's cutoff while holding items fixed, which is the comparison a student-side cutoff cannot provide.
Decisive Checks
The proposal succeeds if three things hold: $\gamma > 0$ with item fixed effects included; $\delta > 0$ under English-only items, so that language is ruled out as the cause; and both effects shrink substantially once we condition on probe accuracy. It is refuted if errors are spread evenly across the four cells, or if errors are concentrated in low-coverage regions but persist regardless of probe accuracy. In the latter case, something other than missing knowledge is at work, such as the socioeconomic projection documented in Large Language Models are Geographically Biased. To separate the two, a final check grades identical correct answers presented with versus without regional framing. A verdict shift in that comparison would indicate regional prejudice in the judge rather than ignorance.
The error types also matter. The single-deficit claim predicts that post-cutoff and low-coverage errors are dominated by accepted fabrications rather than rejected correct answers. We will report the false-acceptance rate on fabricated answers separately for each fabrication type and each cell.
References
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Large Language Models are Geographically Biased
- Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
- How Reliable is Multilingual LLM-as-a-Judge?
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Distributional LLM-as-a-Judge