Large language models are now routinely used as judges: a model reads two candidate answers, writes a critique, and declares a winner. In expert domains the human who checks these verdicts usually reads the judge's written critique rather than redoing the problem. Two findings make that risky. Judges cannot reliably evaluate problems they cannot solve themselves; vanilla GPT-4o is near chance on verifiably hard response pairs (JudgeBench). And fluent natural-language explanations make human experts more likely to accept wrong AI output, by 10.1% among clinicians reading erroneous chest X-ray diagnoses (Fool Me Once?). Put together, the worst case is predictable. On exactly the items where the judge lacks the domain knowledge to solve the problem, it still produces a confident, well-formed critique, and a human overseer accepts it.
We propose to measure this failure directly and to replace the judge's narrative critique with a solve-first certificate. Before giving a verdict, the judge must produce its own independent solution as an executable program. The human sees whether that program runs, what it outputs, and whether its output matches either candidate. The judge abstains visibly when it cannot produce a working solution. The claim is that this turns judging from a matter of trusting a story into a checkable task, the condition under which explanations have been found to help human–AI teams. It should also expose the judge's knowledge gaps to the human instead of hiding them.
Core Claim
Over-reliance on LLM-judge rationales is not spread evenly. It should concentrate on items where the judge cannot solve the underlying problem, because those are the items where its verdict is wrong while its prose stays just as fluent. If that holds, the fix is to show the human the judge's solving ability on each item, not to make the rationale better written. A certificate built from the judge's own executable solution should cut acceptance of wrong verdicts on those items without cutting acceptance of correct ones. It should do better than showing a confidence score, because confidence is itself produced by a model that does not know what it does not know.
The motivation comes from In Search of Verifiability. That meta-analysis of roughly 30 user studies found that human–AI teams outperformed either party alone only when the explanation gave the human a shortcut to independent verification. Those studies used classic decision-support systems, not LLM judges. Two further papers sharpen the problem. The Silent Judge showed GPT-4o and Gemini judges flipping verdicts on provenance and recency labels while acknowledging the cue 0% of the time in their rationales, so the rationale is not a faithful audit trail. Deep Research, Shallow Evaluation found LLM judges more lenient than experts in 85% of citation disagreements, with deeper experts agreeing less with the judge. Neither paper asks what a human does with such a judge's output, and that is our question.
Two papers set limits on the design. JudgeBench establishes the judge–solver coupling we exploit. TheoremQA showed that pasting theorem definitions into prompts does not help. That rules out "give the judge the domain knowledge" as the mitigation and pushes us toward exposing the gap rather than patching it. On method, we borrow Program-of-Thought solving from TheoremQA, where GPT-4 reached 51.0% with it, and the dependency idea from JADE, where downstream reasoning built on refuted premises is invalidated. JADE verifies facts with web search, which cannot check a theorem application. Our certificate instead checks the judge's own computation.
Research Program
Item pool. We need ground truth, so we work in domains where answers can be checked. We take the JudgeBench pairs in Knowledge, Reasoning and Mathematics. We also build a parallel set from TheoremQA questions in math, physics, EE&CS and finance, following JudgeBench's style-neutral recipe: a correct and an incorrect solution generated by the same model, so writing style cannot reveal which is right. To stress the judge we add a subset of JudgeBench-Pro trap items from BiasScope.
Knowledge-gap label. For every item and judge, we run the judge as a standalone solver several times and label the item judge-solvable or judge-unsolvable by majority correctness. This label is never shown to participants. It is the stratifying variable that tests the core claim.
Certificate. The certificate has three parts. First, the judge writes a Program-of-Thought solution. Second, the program is executed in a sandbox. Third, the human sees the code, its output, and a match flag showing which candidate, if either, agrees with the output. If execution fails, or the output matches neither candidate, the interface states that the judge could not independently solve the problem and shows the verdict as unsupported. No narrative critique is shown in this condition.
Judges. We use two judges to separate judge strength from presentation format. One is a mid-strength model, GPT-4o, or a self-rationalizing 8B evaluator trained as in Self-rationalization improves LLM as a fine-grained judge. The other is a strong reasoning judge, o3-mini at high reasoning effort.
Human study. Participants are recruited at two expertise tiers, mirroring the Near-Expert versus Deep-Expert contrast in Deep Research, Shallow Evaluation. The near-expert tier is advanced undergraduates in the relevant field. The deep-expert tier is graduate students or practitioners. Each participant is randomized to one presentation condition and reviews a fixed set of judge verdicts, which includes a controlled share of wrong verdicts drawn from both judge-solvable and judge-unsolvable items. For each verdict the participant accepts or overrides it and reports confidence. The design, the analysis plan and the primary contrasts are preregistered. The Fool Me Once? study with 85 clinicians sets a floor on scale. We will size the study by a power analysis for the interaction between condition and judge-solvability, rather than borrowing a figure.
Metrics. Over-reliance is the rate of accepting wrong verdicts. Appropriate reliance is the rate of accepting correct verdicts. Team accuracy is compared against both the judge alone and the human alone, which is the complementarity test. We also record time per item, since Fool Me Once? reported explanations raising decision time from 36.0s to about 43s per case.
Baselines
Verdict only. The naive reference point is the judge's verdict with no explanation. It sets the floor for over-reliance driven by authority alone. Without it we cannot tell whether a narrative critique adds harm beyond the bare verdict.
Narrative rationale. This is the standard chain-of-thought judge output, the setup in which Self-rationalization reports a 3–9% scoring gain. It is the configuration most deployments actually use and the one our claim targets. The certificate beats it if it lowers wrong-verdict acceptance on judge-unsolvable items while keeping correct-verdict acceptance within a small preregistered margin.
Verdict plus verbalized confidence. This is the cheapest serious competitor. Confidence is elicited from the same judge, on the same items, under the same interface. The certificate must beat it on the reliance gap between judge-solvable and judge-unsolvable items, not only on overall accuracy.
Strong judge with narrative rationale. We run o3-mini at high reasoning effort as the strongest existing automated method. On JudgeBench it reports 80.86% overall accuracy, against 50.86% for vanilla GPT-4o and 64.29% for Claude-3.5-Sonnet (all from JudgeBench). There is no reported result on Lacuna for human over-reliance on any LLM judge's rationale, and none for any judge's accuracy on our TheoremQA-derived pairs, so those numbers are first measured here. The proposal beats this baseline if a mid-strength judge with certificates yields equal or higher human–AI team accuracy than the strong judge with narrative rationale.
Cognitive forcing. The participant solves the item before seeing the verdict. This is the friction-heavy upper reference, and we compare against it on both accuracy and time.
Ablation of the key ingredient. We show the judge's independent final answer and match flag, but no executable code. This isolates whether verifiability matters or whether simply exposing whether the judge could solve the problem does the work.
All conditions use identical items, identical judge calls, and the same per-item time limits.
Why not just…?
Why not just use a stronger reasoning judge? A stronger judge reduces the number of wrong verdicts, but it does not change how humans treat the wrong verdicts that remain. Even strong models break down on trap items: GPT-4o reaches a 74.7% error rate on JudgeBench-Pro (BiasScope). On AppealCase, AppealCase reports reasoning-heavy models not clearly outperforming others, with all models below 50% F1. The strong-judge condition tests this directly. If o3-mini with narrative rationale matches the team accuracy of certificates, the proposal's practical value shrinks to cost savings, and we will report that.
Why not just show a confidence score? This is the most serious threat. If the judge's confidence tracks its solving ability, a single number could do what the certificate does at lower cost. Two pieces of evidence argue against it. Seeing the Reasoning found that confident framing raises adoption even when the reasoning is flawed. In Search of Verifiability found that confidence alone did not yield complementary performance. Our stratified design settles the question. We will measure confidence calibration separately on judge-solvable and judge-unsolvable items. If confidence closes the reliance gap as well as the certificate does, we will conclude that exposing the solve signal is what matters, and that verifiability adds nothing.
Why not just add domain theorems or rubrics to the judge's prompt? TheoremQA found that theorem augmentation "did not provide a significant boost," because the failure lies in applying theorems, not recalling them. Static rubrics also underperformed in JADE. Augmentation might also make the critique more persuasive without making it more correct. For that reason we do not treat it as a mitigation and leave it out of the main arms.
Why not just make humans solve first? Cognitive forcing likely works best on accuracy, but it removes the scalability that is the reason for using a judge, and prior work reports user resistance to it. We include it so we can state the cost honestly. The certificate succeeds if it recovers most of forcing's accuracy at a fraction of its time cost.
Proposed Experiments and Decisive Checks
The first decisive check tests the core claim. In the narrative-rationale arm, the excess acceptance of wrong verdicts over the verdict-only arm should be significantly larger on judge-unsolvable items than on judge-solvable items. If there is no such interaction, the knowledge-gap account of over-reliance is wrong, and the certificate has no specific reason to help.
The second check is the main effect. Certificates should reduce wrong-verdict acceptance on judge-unsolvable items relative to both narrative rationale and confidence display, without a preregistered-margin drop in correct-verdict acceptance. We will also compare the full certificate with the answer-only ablation. If the two perform the same, verifiability is not the operative ingredient.
The third check is complementarity. The certificate condition should produce team accuracy above the judge alone. If it does not, our results confirm the pessimistic reading of In Search of Verifiability even in a domain where answers can be checked.
The fourth check is expertise. We will test whether near-experts benefit more from certificates than deep-experts. That result would matter because near-experts are the people typically hired to audit judges.
Finally, as a boundary test we run a small AppealCase arm. Reversal prediction there has no executable certificate, so the certificate can only list statutes and the facts they are paired with. Following In Search of Verifiability, we expect weak or no gains in this arm. We report it so that the claim covers tasks that can be checked, not judging in general.
References
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges
- Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting
- In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks
- TheoremQA: A Theorem-driven Question Answering Dataset
- JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
- BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
- Self-rationalization improves LLM as a fine-grained judge
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
- Seeing the Reasoning: How LLM Rationales Influence User Trust and Decision-Making in Factual Verification Tasks