Do LLM Judges Grade Earth Science by Textbook Memory? A Decoupling Probe

Language models are now used as automated graders of scientific reasoning. This practice is called "LLM-as-a-judge": a strong model reads a candidate answer or hypothesis and scores it. In Earth observation, one system already uses an LLM judge to rank satellite-data hypotheses by importance, tractability and novelty (EO-Agents). That system never checks whether the judge is right. We propose a test of one specific failure. A judge that has absorbed the textbook story "satellite greenness tracks photosynthesis" may reward explanations that repeat that story and penalize correct explanations that contradict it. That story is the assumption that the Normalized Difference Vegetation Index (NDVI, a satellite measure of how green a canopy looks) moves with gross primary productivity (GPP, the rate at which plants fix carbon). Remote-sensing research has mapped exactly where the story breaks. It holds weakly in arid grasslands and fails almost completely in evergreen needleleaf forests, humid zones, and the shoulders of the growing season. That map gives us a ground truth against which a judge's preferences can be scored.

The note's two-sentence version is this. We build a generator that turns real NDVI, GPP and fluorescence data into paired explanations, one correct and one wrong, where the correct one sometimes agrees with the familiar narrative and sometimes contradicts it. We then measure whether LLM judges fail selectively on the contradicting cases, whether renaming the variables to neutral symbols removes the failure, and whether existing calibration methods catch these errors.

Core Claim

We hypothesize that LLM judges show a consensus-prior bias: their accuracy on scientific explanations depends on whether the true answer matches a widely published narrative, not only on the evidence in front of them. We further claim that this bias has two properties that make it more dangerous than the cue biases studied so far.

The first property is that it is silent. Judges justify the choice with general domain knowledge rather than the numbers provided. The Silent Judge found the same pattern for injected provenance and recency labels, where judges explained their choice in a way that never mentioned the cue (Cue Acknowledgment Rate of 0%).

The second property is that it is confident. Ensemble-based confidence estimates will not flag these errors, because every simulated annotator shares the same memorized prior.

The contamination literature supplies the lens. Post-cutoff probes show models scoring near-perfectly on familiar causal benchmarks but dropping sharply on fresh material, with Claude 3 Opus below 70% on CausalProbe-H (Unveiling Causal Reasoning in LLMs). Our question is whether the same "recall the familiar story" behaviour carries over from the model being tested to the model doing the grading.

What would be new: The Silent Judge audits biases triggered by cues injected into the prompt. EO-Agents measures how far judges disagree with each other, not whether they are correct. Earth-Agent grades Earth-observation agents by matching tool sequences and final values, with no LLM judge involved. None of the three measures a bias that comes from the judge's own background knowledge, against data-derived ground truth, in a domain where that knowledge is known to be wrong in specific places.

Research Program

The probe generator. The generator builds probes from published data products with documented decoupling patterns:

  • GIMMS NDVI and NIRv-derived GPP across China's aridity gradient for 1982–2018. Here coupling between greenness and GPP falls from $R^2=0.24$ in semi-arid zones to $0.11$ in humid zones. By vegetation type it ranges from $0.20$ in grasslands to $0.04$ in evergreen needleleaf forest, and the sensitivity slope falls from $0.12$ to $0.01$ (Decoupling and Insensitivity of Greenness and GPP).
  • MODIS NDVI and GOSIF fluorescence phenology for northeastern China, 2001–2020. Fluorescence is a direct physiological signal of photosynthesis. In this data, NDVI starts the season about 10 days early, ends it about 36 days late, and overstates its length by about 46 days (Vegetation Phenology in Northeastern China).
  • A post-cutoff anomaly: the 2023 land-sink collapse. The atmospheric CO₂ growth rate reached 3.37 ppm while fossil emissions rose only 0.6% (Low latency carbon budget analysis).

What a probe looks like. Each probe is a scenario card. It names a region, vegetation type, aridity class and period, and lists statistics we compute ourselves from these series, such as slopes, $R^2$, and season start and end dates. A question follows, for example "Which of these two sites' carbon uptake is better tracked by greening?" or "How long was the photosynthetically active season?". Two candidate explanations come with it. We use comparative questions on purpose. Even the "coupled" regimes have modest $R^2$, so absolute thresholds would be arbitrary, while "site A is more coupled than site B" has an unambiguous answer in the data.

Controls inside each probe. The two explanations are matched in length, structure and tone, and differ only in their conclusion and mechanism. Each probe is shown in both orders. A probe is labelled narrative-consistent when the true answer agrees with "greenness tracks carbon," and narrative-inconsistent otherwise.

Generating many probes. Following the procedural idea in DyVal, the generator samples site pairs, years and question templates. It also varies a depth parameter: the judge receives either the summary statistics or the raw time series from which it must infer them. This yields thousands of probes rather than one fixed set. The generator itself sits in a private repository, so the probe instances can be refreshed if released probes leak into training data.

The renamed twin. This is the key ingredient. For every probe we produce a twin with identical numbers and reasoning structure, but with the domain stripped out. NDVI becomes "index X," GPP becomes "flux Y," and vegetation types become "class 1/2." If a judge is right on the twin and wrong on the real-named probe, the error comes from domain prior rather than inability to read the evidence.

We add a familiarity covariate. We compute Min-K% Prob, a test that flags text a model assigns unusually low perplexity to, on each correct and incorrect explanation using open-weight judges. We then test whether judge errors concentrate where the wrong explanation looks familiar.

Baselines

Naive reference. A rule that always picks the narrative-consistent explanation, plus random choice. The rule scores 100% on consistent probes and 0% on inconsistent ones by construction. A judge whose accuracy gap between the two categories approaches that of this rule is effectively following the textbook story.

Plain LLM judge. GPT-4o, Claude, Gemini and an open-weight model are run zero-shot with evidence in context, at temperature 0 and at a realistic sampling temperature, using one shared prompt template tuned only on a held-out development split. This is the configuration deployed in EO-Agents, so it is the default users rely on. We have no prior accuracy on our probes; we found no reported result on Lacuna for LLM-judge accuracy on data-grounded NDVI–GPP explanations. As a related reference point, GPT-4o reached only 57.8% accuracy tagging errors in math reasoning traces (Local Coherence or Global Validity?).

Strongest existing method. The confidence-calibrated escalation cascade of Trust or Escalate. It estimates judge confidence by simulating several annotator personas and passes uncertain cases up to a stronger model or a human. On AlpacaEval, Chatbot Arena, TL;DR and Auto-J it reports about a 50% reduction in Expected Calibration Error, and over 80% guaranteed human agreement at nearly 80% coverage. It is the strongest competitor because it is the only method in the file that promises formal agreement guarantees, and it would remove our concern if it routed consensus-prior errors to escalation. We run it with its published persona count and risk level on the same probes, calibrating its thresholds on our development split with the same budget as the plain judge. It counts as beaten if, at matched coverage, its accuracy on narrative-inconsistent probes stays significantly below its accuracy on consistent ones. It is also beaten if its guarantee fails on the inconsistent subset while holding on the consistent subset.

Isolating ablation. The renamed twin. The accuracy difference between real-named and renamed versions, computed separately for consistent and inconsistent probes, is the quantity the proposal is about.

Oracle reference. A programmatic checker that recomputes the statistic and is correct by construction. It bounds what any judge could achieve.

Why not just…?

Why not just grade with programmatic checks and drop the LLM judge? For our probes we can, and the checker is our oracle. The judge matters because real deployments use it on open-ended hypotheses where no checker exists, as in EO-Agents across roughly a million dataset pairings. Our probes are the rare setting where a judge's habits can be measured against truth before trusting it where truth is unavailable. This alternative does not answer the question; it supplies our ground truth.

Why not just put the evidence in context and assume the judge reads it? This is a real threat. EO-Agents found that showing operational metadata measurably changed tractability ratings, so judges do react to context. If gaps vanish whenever summary statistics are present, the bias is a non-issue for well-specified prompts, and we will report that.

Our design tests this directly. Every probe already contains the evidence, and the depth parameter varies how much inference is needed to use it. The bias claim survives only if the inconsistent-minus-consistent gap persists with evidence present and grows with depth. DyVal observed this kind of decay as tasks required more sequential steps, with GPT-3.5 dropping 23% across difficulty tiers.

Why not just ask for chain-of-thought or verbalized confidence? The evidence in the file is against it. Justifications in The Silent Judge never revealed the shortcut. Verbalized confidence calibrated poorly in Trust or Escalate. Plain chain-of-thought did not overcome memorization on post-cutoff causal probes (Unveiling Causal Reasoning in LLMs). We still include a chain-of-thought arm. We code whether each rationale cites the provided numbers or appeals to general knowledge, which gives an evidence-acknowledgment rate analogous to Cue Acknowledgment Rate.

Why not just use a static held-out set of post-2023 scenes? A fixed fresh set would detect the bias once. However, it cannot vary inference depth, and it loses protection once published, the same decay the CausalProbe authors face. We include the 2023 sink anomaly as a static post-cutoff slice and compare it with generated probes. If static and generated probes give the same gaps, the generator's added value is only refreshability, and we will say so.

Proposed Experiments and Decisive Checks

Primary measurement. For each judge we report accuracy on narrative-consistent and narrative-inconsistent probes, for real-named and renamed versions, with order-swap consistency and bootstrap confidence intervals. The 2×2 design (consistent vs. inconsistent × real-named vs. renamed) is the core analysis.

Decisive result for the claim. An interaction in which the real-named inconsistent cell is markedly worse than the other three cells, while renamed accuracy is similar across both categories. If renamed probes show the same gap, the failure is plain reading difficulty, not memorized priors, and the claim is falsified.

Secondary tests. First, whether per-probe error rises with Min-K% familiarity of the wrong explanation. Second, whether the gap is largest for the most decoupled regimes, namely needleleaf forest and phenology shoulders. Third, whether Trust-or-Escalate confidence on wrong inconsistent-probe verdicts is as high as on correct verdicts; that is the signature of shared, confident priors.

Downstream check. We re-score a sample of EO-Agents-style hypotheses that propose NDVI as a carbon proxy in decoupled biomes. A small panel of remote-sensing experts rates them, and we test whether judge leniency on these hypotheses tracks the probe-measured bias.

Possible mitigations. If the bias is confirmed, we test two:

  • Requiring the judge to quote the relevant statistic before deciding.
  • Using a renamed-twin consistency check as an escalation trigger: escalate whenever real-named and renamed verdicts disagree.

Each is compared against the Trust-or-Escalate cascade at matched cost.