Single-Substitution Probes: Measuring and Repairing Mutation Blindness in LLM Judges

When we let a large language model grade another model's open-ended output — "LLM-as-a-judge" — we assume the judge actually reads the content it scores. The evidence says it often does not: on ELI5, GPT-4o flipped its verdict on 30% of pairs when a response was relabeled "NEW" instead of "OLD," and when the provenance tag was swapped between "EXPERT" and "UNKNOWN" the verdict shifted 18% of the time, yet the judge cited the cue in its written justification exactly 0% of the time, fabricating content-based reasons instead (The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge). A structurally identical failure exists in computational biology, one level down: models asked to predict how a protein binds its target barely react when a single amino acid in the protein is swapped, producing predictions that correlate near 1.0 with their wild-type predictions even for drug-resistant mutants (Towards Precision Protein-Ligand Affinity Prediction Benchmark: A Complete and Modification-Aware DAVIS Dataset); AlphaFold2 likewise folds mutated sequences into nearly the wild-type backbone (Kabir et al.), and global structural similarity cannot separate pathogenic from benign BRCA1 variants (Li et al.).

This proposal treats these as one problem seen through two lenses, and exploits the overlap. Single amino acid substitutions are the rare perturbation for which we have minimal edits with laboratory-measured consequences — some swaps flip antibody binding entirely (Tsuruta et al.). That makes them an ideal microscope for judge reliability: construct twin evaluation items, identical except for one substitution that flips which answer is physically correct, and ask whether the judge's verdict flips with it. If it does not, the judge is mutation-blind, and its mutation-specific justifications are fabricated. We then use measured blindness as the trigger for a compute-aware escalation policy, so expensive frontier judges are spent only on items where cheap judges are demonstrably insensitive — a direct extension of the selective-escalation machinery in Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement, which escalates from Mistral-7B to GPT-4 but triggers on confidence rather than on demonstrated insensitivity to content.

Core Claim

The claim has three parts. First, frontier LLM judges asked to compare natural-language scientific claims about wild-type versus point-mutant binding will be substantially mutation-blind: their verdicts on twin items will agree far more often than the measured affinity differences warrant, echoing the near-1.0 wild-type correlation documented for sequence-based binding models on the modification-aware DAVIS benchmark (Towards Precision Protein-Ligand Affinity Prediction Benchmark). Second, this blindness will coexist with fluent, mutation-specific justifications — a direct analogue of the 0% Cue Acknowledgment Rate in The Silent Judge, where judges never mention the cue that actually drove the verdict. Third, an escalation policy triggered by a cheap twin-probe — rather than by the judge's own stated confidence — will recover most of the frontier judge's discrimination at a small fraction of its cost, because Trust or Escalate showed LLMs are overconfident when verbalizing confidence, which is precisely the wrong signal to gate escalation on when the failure mode is insensitivity.

Two honest caveats about grounding. The Lacuna dossier was truncated partway through the judge-bias section, so the prompt-sensitivity and computational-cost directions are represented here only through the papers that survived the cut; and no reported result found on Lacuna for any existing minimal-edit, ground-truth-controlled sensitivity probe of LLM judges. The closest existing works are The Silent Judge, which varies surface metadata (dates, provenance tags) rather than the substantive content, and Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs, which shows generalist LLMs fail on raw protein sequences but recovers performance (87.19 on the Protein QA Benchmark for Gemini 2.5 Pro) by replacing sequences with tool-derived textual context — a task-performance study, not a study of judge verdicts or bias. Ours differs in both the perturbation (a semantically decisive edit with physical ground truth, not a metadata label or a formatting change) and the object of study (the judge's verdict dynamics and their repair, not the model's task accuracy).

Research Program

We build three things. The first is a paired-item benchmark, which we call twin-item panels, assembled from two sources in the file: modified kinase variants with measured affinities from the modification-aware DAVIS dataset (Towards Precision Protein-Ligand Affinity Prediction Benchmark), and SARS-CoV-2 neutralization data of the kind used by Multi-Modality Representation Learning for Antibody-Antigen Interactions Prediction. Each item presents two short natural-language reports making conflicting claims about binding outcome; one is correct. Its twin is byte-identical except that the single substitution known to flip the ground truth is applied. The second is a measurement protocol: Verdict Shift Rate on ground-truth flips (adapted from The Silent Judge), a Mutation Acknowledgment Rate scoring whether the justification names the residue change, and a paraphrase grid of roughly twenty instruction rewrites per item to measure how much mutation sensitivity itself wobbles under prompt change. The third is the escalation policy: a cheap judge handles every item, and the item escalates to the frontier judge exactly when the cheap judge fails the local twin-probe,

$$\text{escalate}(i) \iff J_{\text{cheap}}\!\left(i^{\text{WT}}\right) = J_{\text{cheap}}\!\left(i^{\text{mut}}\right),$$

where $J$ returns the verdict on an item and the two items differ only by the decisive substitution. Statistical guarantees on final accuracy follow the fixed-sequence-testing approach of Trust or Escalate, replacing its confidence trigger with this sensitivity trigger.

Proposed Experiments

Experiment 1 quantifies mutation blindness. Four judges spanning cost tiers (including GPT-4o and Gemini 2.5 Pro, the models named in the file) each score the twin panels under a fixed default prompt, under raw-sequence prompts, and under context-rich prompts built from domain and homology annotations, following the prompt construction of Lost in Tokenization, which found that supplying the raw sequence alongside context actually degraded performance. We report Verdict Shift Rate on flipped ground truth and Mutation Acknowledgment Rate, and we run the same instrument on the ELI5 metadata-swap setting of The Silent Judge to confirm the probe catches the known bias there too. A panel-vote arm (majority across judges) tests whether voting cures what individual judges lack; Investigating Non-Transitivity in LLM-as-a-Judge predicts it will not, since averaging circular preferences cannot create sensitivity to an edit every voter ignores.

Experiment 2 measures brittleness of the sensitivity itself: across the paraphrase grid, we report the variance of mutation sensitivity per judge. The expectation, grounded in Lost in Tokenization, is that raw-sequence prompts collapse sensitivity while context prompts partially restore it, and that no single instruction stabilizes it across judges.

Experiment 3 evaluates the escalation policy on accuracy-versus-cost. Accuracy is against the measured ground truth (not human preference), cost is the fraction of items escalated to the frontier judge, and calibration is reported as Expected Calibration Error, the metric Trust or Escalate reduced by roughly 50% using K-shot annotator simulation. As a signal-extraction reference, we report the specialized model's numbers from the file — a Pearson correlation of 0.819 on wild-type affinity and a ROC-AUC of 0.757 on SARS-CoV-2 neutralization for MuLAAIP (Multi-Modality Representation Learning for Antibody-Antigen Interactions Prediction) — to show how much physical signal exists in these panels; no reported result found on Lacuna for any LLM judge's discrimination on mutation-flip items, which is the gap this measures.

Baselines

The naive reference point is a single frontier judge with one fixed, hand-tuned prompt — the "do nothing" practice that underlies judge-based benchmarks such as MathHay. It is the strongest competitor on accuracy-per-call and the weakest on cost and on diagnosability; we run it on identical items with the same prompt-tuning budget as every other arm, and beating it means matching its accuracy while escalating on at most a third of items. The strongest existing method is the Trust or Escalate pipeline: K-shot annotator simulation to calibrate a cheap judge, selective escalation from Mistral-7B to GPT-4, and fixed sequence testing for an above-80% human-agreement guarantee — the paper reports roughly a 50% Expected Calibration Error reduction from the simulation step. We reimplement it faithfully and run it on the same twin panels with the same compute and the same number of allowed frontier calls; beating it means higher accuracy on ground-truth-flip items at equal or fewer frontier calls, with the decisive comparison on the subset where the ground truth flips but the judge's confidence stays high — exactly the items a confidence trigger should miss. The key-ingredient ablation is the same escalation machinery with the twin-probe trigger replaced by verbalized confidence, holding everything else fixed; if the twin-probe does not beat the confidence trigger on that subset, the proposal's central mechanism is not earning its keep.

Why not just…?

Why not just use the specialized binding model instead of a judge? Multi-Modality Representation Learning for Antibody-Antigen Interactions Prediction and GeoPPI already predict mutation effects from structure and sequence, and on pure affinity prediction they would win; we do not claim to beat MuLAAIP's 0.757 neutralization ROC-AUC and will report it as a reference ceiling. But they cannot read or grade a free-text scientific report, which is what judge pipelines exist for, and they inherit the upstream wild-type bias of structure predictors documented by Kabir et al.. The honest residual threat is that the community mostly wants binding numbers, not judged prose — in which case this proposal's audience is the judge-reliability community, and the biological panels are its instrument rather than its product.

Why not just fix the prompt? Lost in Tokenization shows prompt composition swings protein-task performance enormously (87.19 with tool context for Gemini 2.5 Pro, degraded when raw sequences are added), so a skeptic will expect a well-engineered context prompt to solve everything. Experiment 2 is designed to answer this directly: if one prompt construction yields high and stable mutation sensitivity across judges and paraphrases, that is a cheaper answer and we will say so. We doubt it, because The Silent Judge shows the distortion survives content-focused prompting and is never acknowledged — a judge can be blind while sounding attentive, which per-prompt fixes cannot detect.

Why not just escalate on low confidence, as Trust or Escalate already does? That is the strongest simple alternative and it is already built; our claim is only that its trigger is miscalibrated for this failure mode, since the same paper shows judges are overconfident when verbalizing confidence, and an overconfident blind judge escalates nothing. The confidence-triggered ablation in Experiment 3 settles this empirically rather than by argument: if confidence and the twin-probe flag the same items, the simpler policy wins and we adopt it.

Decisive Checks

The core claim is falsifiable up front: if Experiment 1 shows frontier judges already flip their verdicts correctly on more than, say, 90% of ground-truth flips under a default prompt, then judges are not mutation-blind, the justification-fabrication claim fails with it, and the escalation layer solves a non-problem — we would report that negative result, which would itself be the first ground-truth-controlled measurement of judge content-sensitivity on Lacuna. The proposal succeeds if judges prove blind (verdict agreement far above what measured affinity differences allow), if the blindness hides behind fluent justifications at a near-zero acknowledgment rate, and if the twin-probe trigger recovers frontier-level accuracy on flip items at a bounded escalation fraction — with the pre-registered target of at most one frontier call in three.