When a language model acts as a judge, grading whether another model's answer is true, its written explanation often has little to do with what actually decided the verdict. If you add a label like "EXPERT" or "NEW" to one answer, the judge's verdict shifts. In pairwise comparisons, GPT-4o shifts by up to 30%. Yet the judge never mentions the label and justifies its choice entirely on content (The Silent Judge). Bold text and lists sway judges the same way (From Lists to Emojis). The judge's rationale and its verdict are misaligned, just as a reasoning model's chain of thought can be disconnected from its final answer. Hallucination detectors trained on such judges inherit the problem.
We propose to remove the judge's freedom to produce a verdict that is not backed by checkable evidence. The judge is forced, at decoding time, to split the candidate answer into consecutive claim spans that together cover all of it. For each claim, it must either quote a verbatim span from the source document or declare the claim unsupported, and then label the claim. A fixed rule outside the model turns these labels into the verdict. Under this design a biasing cue can only change the outcome by changing specific claim labels, and each such change is visible and auditable. The judge also cannot invent evidence, because every quotation is copied from the source by construction.
Core Claim
Our claim is that most shortcut and format bias in hallucination judging lives in the step where the judge produces a free-form verdict and a free-form justification. If the judge's output is constrained so that the verdict is a deterministic function of source-anchored, full-coverage claim labels, most of that bias should disappear. What remains becomes measurable as "label shift": the rate at which a cue flips individual claim labels.
We do not claim that constraints make the judge smarter. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning and SynCode both stress that grammars guarantee syntax, not semantics. Our claim is narrower. The constraint closes three specific escape routes:
- Fabricated evidence. The judge cannot quote evidence that does not exist.
- Selective omission. The judge cannot skip the claims that would count against the answer it favors.
- Verdict independence. The judge cannot state a verdict that does not follow from its own claim-level analysis.
How This Differs From Prior Work
Work that motivates the proposal. Two papers found the gap.
- The Silent Judge shows verdict shifts with a 0% rate of the judge acknowledging the cue. It only measures prompt-injected metadata on free-form judges and does not intervene on the output structure.
- Local Coherence or Global Validity? shows GPT-4o agrees with human error tags on reasoning traces only 57.8% of the time. The judges that audit reasoning are themselves unreliable.
Work that supplies the method. The input-dependent grammars of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning build the allowed output language from the input itself. That paper used them to restrict entity outputs to a 2.7-million-entry candidate set. We build the grammar per example from the candidate text and the source document, using the Transformers-CFG implementation in epfl-dlab/GCD.
Closest existing judges. Three lines of work structure the judge's reasoning, but all of them leave both the verdict and the evidence as free text.
- Halu-J trains a 7B judge to follow a four-stage critique.
- PlanJudge, in Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases, prompts the judge to write a rubric first.
- JudgeLRM rewards
<think>/<answer>tags during training.
In all three, the model still writes the final verdict, and nothing checks that its quotations exist. We keep the model's role in segmenting and labeling claims and take the verdict away from it.
Work that constrains the design. Two papers limit what this design can do.
- Structured Hallucination in Tool-Using Agents shows that forcing models to emit structure can itself corrupt 11–13% of table cells. Our constrained fields are therefore copy-only spans and small label sets, never free generation of values.
- Deng's cognitive-load argument predicts that heavy syntax constraints can degrade reasoning. We test this directly rather than assume it away.
Research Program
The grammar. For each item we compile a grammar with three parts:
- Claim tiling. The candidate response is split into an ordered sequence of claim spans. Each span must be a contiguous substring, and together the spans must cover the whole response, with formatting markup such as bold and bullets stripped from the claim text.
- Evidence. Each claim takes either a verbatim substring of the source document or the token
NONE. - Label. Each claim takes one label from supported, contradicted, or unverifiable.
The tiling constraint forces the judge to consider every claim in the response. The copy constraint means that quoted evidence always exists in the source.
The verdict rule. For single-answer grading, a response is marked hallucinated if any claim is labeled contradicted, or if any claim is labeled unverifiable while carrying substantive content. For pairwise comparison, the response with fewer non-supported claims, weighted by span length, wins, and ties are declared rather than broken by the model. This rule is fixed before any experiment is run.
Application to reasoning traces. We apply the same design to math traces on GSM8K. There, the "source" for each step is the problem statement plus earlier steps. Numeric claims additionally carry a constrained arithmetic expression, which a calculator checks.
Models. The method needs access to output logits, so judges are open models at 7B–32B scale. Closed APIs appear only as free-form reference points.
Baselines
Naive free-form judge. The do-nothing reference is the same open judge model prompted to read the source and candidate and output a verdict with a justification. Because it is the same model, any difference is attributable to the output structure. We also report GPT-4o as a free-form judge where the benchmark already has its numbers. On ME-FEVER, Halu-J reports 83% accuracy for GPT-4o.
Halu-J, the strongest hallucination judge in the file. It reaches 91% accuracy on ME-FEVER, with a 68.11% evidence-matching rate. We run it with its released four-stage format and also use it as the backbone for our grammar. This separates the gain from training from the gain from constraining the output. Beating it means two things:
- matching its 91% accuracy within noise, and
- substantially lowering cue-induced verdict shifts and reaching 100% verbatim evidence, where Halu-J's quotations are checked post hoc.
PlanJudge, the strongest prompt-only debiasing method. It is the natural competitor on bias. It lifts DeepSeek-R1 on BiasBench from 65.00 to 97.50 (Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases). We run it on the same backbone models.
Fair comparison. All methods see identical prompts, sources, and cue injections, and all receive the same decoding token budget. Any method allowed to resample receives the same number of samples.
Ablations isolating the key ingredients. The proposal has two ingredients: enforcing the structure at decoding time, and computing the verdict outside the model. Three ablations separate them:
- Prompted schema. The identical JSON structure is requested by prompt only. Invalid or fabricated spans are rejected and resampled up to the same budget. This isolates decoding-time enforcement from the schema itself.
- Constrained plus model verdict. The same grammar is used, but the model writes its own verdict field at the end. This isolates the external verdict rule.
- Constrained without tiling. The judge may pick which claims to extract. This isolates the full-coverage requirement, which is our defense against selective omission.
Why not just…?
Why not just prompt the judge to quote evidence in JSON and check the quotes afterward? This is the most serious threat. Strong models follow JSON prompts much of the time. Still, Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning found unconstrained LLaMA produced valid structures only about 64% of the time, and prompting cannot enforce tiling or verbatim copying during generation. The prompted-schema ablation, with an equal resampling budget, settles the question. If it matches our cue robustness and accuracy at similar cost, we will report that the contribution is the claim-tiled, externally computed verdict, and that decoding constraints are merely a convenience.
Why not just ask for a rubric first (PlanJudge)? PlanJudge's large BiasBench gain concerns length and verbosity bias measured with the verdict still model-generated. It does not show that the rubric actually governs the verdict. The Silent Judge shows fluent plans and justifications can conceal the real cue. The cue-injection experiment below measures whether PlanJudge's verdicts still shift under provenance and recency labels that its rubric never mentions.
Why not fine-tune a better judge, as Halu-J does? Supervised training hits a ceiling on reasoning-heavy items (JudgeLRM). Preference data also bakes in format shortcuts: 0.7% biased data moved a reward model's list preference from 51% to 77.5% (From Lists to Emojis). More importantly, fine-tuning and constraints are complementary. Running the grammar on the Halu-J backbone shows whether the two add up or whether training already captures the benefit.
Why not just take a majority vote over several judge samples? Every sample sees the same cue, so voting reduces noise but not a shared systematic shift. Format-Adapter reports that same-format self-consistency converges on shared blind spots. We include an 8-sample vote, which costs more tokens than our single constrained pass, and compare verdict shift rates directly.
Why not diff the output against the source programmatically? For pure copy tasks, such as rendering flight API results, this is correct and strictly better. The metadata-injection bypass in Structured Hallucination in Tool-Using Agents reaches 100% fidelity. We therefore use deterministic diffing as an upper bound on that benchmark and target the setting it cannot handle: paraphrased, summarized, or filtered answers whose truth requires semantic matching.
Proposed Experiments
Accuracy. We measure judge accuracy on ME-FEVER against Halu-J's reported 91%. We also measure it on HaluEval QA, dialogue, and summarization, where ChatGPT's reported detection accuracy is 62.59%, 72.40%, and 58.53% (HaluEval). For WANLI, ANLI, and KBQA, no reported result was found on Lacuna, so we report our own runs of all baselines.
Bias. We port the Silent Judge protocol to HaluEval, whose items pair a correct response with a hallucinated one, making pairwise judging natural. We inject EXPERT/UNKNOWN and NEW/OLD labels in both orders, and separately add bold text and lists to the hallucinated response. Three metrics are reported:
- Verdict shift rate, following The Silent Judge. GPT-4o's +30% on ELI5 is the reported reference for free-form judges.
- Claim label shift rate, our residual measure of bias.
- Evidence fabrication rate for the free-form baselines: the fraction of quoted evidence that does not appear verbatim in the source.
Reasoning traces. We rerun the human-tagged GSM8K error analysis of Local Coherence or Global Validity?, where GPT-4o reached 57.8% agreement, with our step-level variant that includes premise pointers and calculator-checked arithmetic.
Structured outputs. On the flight benchmark, we test whether constrained judges catch the value mutations and silent truncations that make up the reported 11–13% cell corruption, with deterministic diffing as the upper bound.
Decisive checks. Five outcomes decide the proposal:
- Main result. The proposal succeeds if the constrained judge cuts verdict shift under cues by at least half relative to the same model judging free-form, while staying within two points of its accuracy.
- Cost of constraints. If accuracy drops sharply, Deng's cognitive-load objection is confirmed for judges.
- Where bias goes. If label shift rises to match the removed verdict shift, bias has only moved into the claim labels. That is still an auditable result, but it is not a fix.
- Coverage ablation. If tiling makes no difference, omission was not the mechanism.
- Enforcement ablation. If the prompted-schema ablation performs equally well, decoding-time enforcement is unnecessary. We would report exactly that.
References
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- From Lists to Emojis: How Format Bias Affects Model Alignment
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- SynCode: LLM Generation with Grammar Augmentation
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Halu-J: Critique-Based Hallucination Judge
- Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases
- JudgeLRM: Large Reasoning Models as a Judge
- Structured Hallucination in Tool-Using Agents: Measuring and Mitigating LLM Synthesis Corruption in Production
- Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models