Do Hallucination Judges Grade the Tone Instead of the Facts?

Language models are now routinely used as judges that decide whether another model's answer is factually correct. These "LLM-as-a-judge" setups decide which outputs count as hallucinations, meaning confident statements that are false or unsupported by the evidence. This proposal asks a simple question. Hold the factual content of an answer fixed. Then change only its emotional tone toward a person or policy and the side it takes (its "stance"). Does the judge's verdict on factual correctness change?

Stance-detection research has shown that models confuse tone with position. A negative tone toward a failed regulation often signals support for reform, yet models routinely read negative wording as opposition. Work on judges has shown that they flip verdicts on irrelevant cues and never admit it. We think the two failures combine. A factuality judge may be more lenient toward false claims wrapped in the "right" framing, and harsher toward true claims wrapped in the "wrong" one, while writing a justification that cites only the evidence. To test this cleanly, every framed rewrite must provably keep the same facts. Fully automatic rewriting cannot guarantee that, so humans verify the rewrites. Their verified variants then serve twice: first as the measuring instrument, then as the training signal for a judge that is unaffected by framing.

Core Claim

We claim three things. First, factuality judges show a measurable framing shift: their true/false verdicts move when only sentiment and stance change. Second, the shift is directional, tracking the judge's prior attitude toward the target. Third, it cannot be removed by prompting or by self-consistency training. It can be reduced by preference-tuning on human-verified framing variants, and this works better than tuning on the same variants left unverified.

The diagnostic contribution stands on its own whatever happens to the mitigation. The mitigation claim is falsifiable against a strong preprocessing baseline described below.

Relation to Prior Work

Three papers motivate the gap.

The Silent Judge showed that GPT-4o and Gemini-2.5-Flash shift pairwise verdicts on provenance and recency cues. The largest shift was +30% on ELI5 for GPT-4o. Their Cue Acknowledgment Rate, the fraction of justifications that mention the cue, was 0%. That work tested labels attached to answers, not framing inside the text. It also tested quality judgments, not evidence-grounded factuality.

Counterfactual Augmented Calibration (FACTUAL) documented sentiment-stance spurious correlation and target preference bias in LLM stance classifiers. Its "Bias-SSC" score for GPT-3.5 on SemEval-2016 is 27.13. FACTUAL reduces this with LLM-generated counterfactual rewrites, but its authors flag that those rewrites may themselves hallucinate. We reuse its rewrite design, apply it to judges rather than classifiers, and put humans in the loop to close exactly that hole.

SHARP showed that a model's stance toward a speaker warps its factual agreement in role-play. It reports sycophancy and adversarial behavior correlated at −0.67. We move this from fictional personas to a judge evaluating political content.

Two papers supply methods and the main competitors.

Halu-J trains a 7B critique-based factuality judge with preference optimization. Direct Preference Optimization (DPO) tunes a model to favor chosen over rejected outputs. Halu-J reaches 91% accuracy on ME-FEVER, compared with 83% for GPT-4o.

ConsJudge tunes judges toward agreement across prompts, using no gold labels. Our concern is that a judge biased the same way under every prompt is perfectly consistent and still wrong. ConsJudge would reinforce that bias.

For routing scarce human effort, we borrow the committee-disagreement idea from SQBC. In that work, humans labeled only the 25% of real samples on which synthetic neighbors disagreed. This reached 0.754 F1 on X-Stance against 0.727 for full real supervision. MPVStance and RATSD constrain the design. Both treat stance and truthfulness as separable, and RATSD explicitly separates "affirming a claim is true" from "favoring the topic." Our framings must vary the second while holding the first fixed.

Research Program

Seed items. We take claims with known truth labels and supporting evidence from two sources. The first is TSD-CT, which provides 3,105 PolitiFact claim-tweet pairs. The second is ME-FEVER, which provides 3,901 claims with noisy, contradictory, and misleading evidence. We add targets and naturally framed texts from SemEval-2016 Task 6 and P-Stance.

Framed variants. For each claim, an LLM writes a short answer that asserts the claim. It then writes variants in a two-by-two design: positive or negative sentiment toward the target, crossed with a favor or against stance. The factual assertion stays unchanged. A fifth variant is a neutral paraphrase with no change in framing. It measures how much verdicts move from rewording alone, which sets the noise floor.

Human verification and routing. Annotators check each variant on two questions. Is the factual assertion identical in content, hedging, and entities? Does the variant realize the intended framing? To avoid the reviewer fatigue documented by BrokenMath, a committee screens variants first. Three rewriting models and an automatic claim-equivalence checker each give a verdict. Variants on which the committee disagrees go to humans first. The rest are spot-checked at a fixed rate. We report the rate at which unverified rewrites altered the facts. This directly measures the risk FACTUAL left open.

Measurement. Let $v_J(x)$ be judge $J$'s factuality verdict on text $x$, given the evidence. For claim $i$, let $x_i^{f}$ be its framed variants and $x_i^{p}$ its neutral paraphrase. We define the framing shift as

$$\mathrm{FS}(J) = \Pr_i\big[\exists f: v_J(x_i^{f}) \neq v_J(x_i^{0})\big] - \Pr_i\big[v_J(x_i^{p}) \neq v_J(x_i^{0})\big].$$

Here $x_i^{0}$ is the original answer. The second term subtracts plain rewording sensitivity, so FS isolates framing. We also measure a directional effect: the gap in false-claim acceptance between framings that agree and disagree with the judge's prior attitude toward the target. That prior is estimated with the target-preference probes from FACTUAL. Finally, we compute the Cue Acknowledgment Rate on the judges' rationales.

Mitigation. We fine-tune a Mistral-7B judge with DPO, following the Halu-J recipe. The chosen response is the correct verdict, held constant across verified variants of a claim. The rejected response is the framing-swayed verdict the base judge actually produced. Training uses some targets, and evaluation uses held-out targets.

Baselines

The naive reference point is an untouched judge: GPT-4o and the base Mistral-7B, prompted with the same rubric and the same evidence. Its FS sets the size of the problem.

The strongest existing factuality judge is Halu-J. It reports 91% accuracy on ME-FEVER, while GPT-4o reports 83% (Halu-J). It is the bar for accuracy on unframed inputs. Our tuned judge must stay within one point of Halu-J's ME-FEVER accuracy, as reproduced in our setup, while lowering FS. We found no reported framing-shift number on Lacuna for any judge.

ConsJudge is the strongest competitor that trains a judge for robustness without labels. We run it on the same seed claims and the same base model with the same DPO budget. To make the comparison fair, the multiple prompt templates ConsJudge needs come from the same rewriting models. Beating it means a lower FS on held-out targets at equal accuracy.

The key ablation isolates human verification. It uses FACTUAL-style unverified variants, with the same count, same base model, same DPO budget, and same hyperparameter search. If verified data does not beat unverified data on FS, the human step adds nothing.

A routing ablation compares committee-routed verification with random routing at the same human budget.

Why not just…?

Why not just tell the judge to ignore tone? FACTUAL reports that explicit debiasing prompts fail for stance because the bias sits in the weights. That finding was for classifiers, not judges, so we include the prompt as a baseline and report its FS directly.

Why not just neutralize or decompose the text before judging? This is the real threat. An LLM could strip the framing, or extract framing-free atomic claims, before the judge sees anything. If that brings FS to the paraphrase floor at equal accuracy, it is the better engineering answer. The contribution would then reduce to the benchmark and the measured fact-alteration rate of the neutralizer, and we would say so.

We suspect neutralization will fall short for two reasons. The neutralizer is itself a model exposed to the same target preference, so it may soften or drop claims selectively. It also adds a second hallucination point. Our human verification labels let us measure exactly how often neutralization changes facts, broken down by framing condition.

Why not just use self-consistency, as ConsJudge does? A directional bias that is stable across prompts yields high consistency. The comparison above settles empirically whether consistency training reduces FS or leaves it intact.

Why not just skip the humans and use more synthetic pairs? SQBC suggests that targeted human labels on ambiguous cases beat volume. The verified-versus-unverified ablation tests this directly. We additionally run it at twice and four times the unverified volume.

Decisive Checks

The diagnosis holds if FS is reliably above zero for GPT-4o and Halu-J-style judges on TSD-CT and ME-FEVER variants. Reliability means bootstrap intervals that exclude zero. The diagnosis also requires the directional gap in false-claim acceptance to have a consistent sign aligned with measured target preference. A CAR near zero would extend the Silent Judge finding to factuality judgments.

The mitigation holds if the verified-DPO judge has lower FS on held-out targets than the prompt, ConsJudge, unverified-data, and neutralization baselines. It must also keep ME-FEVER and TSD-CT accuracy within one point of the strongest baseline.

If FS turns out to be indistinguishable from the paraphrase floor, judges are not fooled by framing. That negative result would itself be worth publishing, because it would justify using LLM judges on polarized content.