Blind Verifiers for Bias-Robust RAG Hallucination Judging

Retrieval-augmented generation (RAG) means a language model answers using documents fetched from an external corpus. The standard way to check whether its answer made things up is to hand the documents and the answer to another model, an "LLM judge," and ask for a verdict. The trouble is that these judges are swayed by things that have nothing to do with truth. They prefer responses labelled as coming from experts or dated recently, and they prefer longer and more polished text. They also never mention these cues in their explanations: The Silent Judge measured a cue acknowledgment rate of 0% for GPT-4o and Gemini-2.5-Flash. Because such a judge's explanation will not reveal the bias, reading its reasoning is no defense.

We propose a structural defense instead. The judge is split into a planner and a verifier. The planner breaks the answer into small, self-contained claims. A separate verifier call then checks each claim against the retrieved evidence while seeing nothing else: not the full answer, its style, its metadata, its position, or any competing answer. A fixed rule, not a model, combines the per-claim verdicts. In two sentences: a judge cannot be biased by what it never sees, so we build a hallucination judge whose deciding component is blind to everything but one claim and its evidence. We then measure, using controlled cue injection, how much judge bias this blindness removes and at which stage any remaining bias enters.

Core Claim

Earlier work split verification into steps to catch more errors. Here the split serves as an information barrier against judge bias, and we test that property directly. The closest prior systems each have part of the design but never test it for bias. RT4CHART decomposes RAG answers into atomic sentences, verifies them locally and globally against context, and aggregates pessimistically. Its global stage re-reads the full context with hints, and it was never tested for shortcut bias. Halu-J orders its judging into evidence categorization, reordering, analysis, and aggregation, but a single model sees the whole response throughout. Chain-of-Verification and FactPrompt show that answering verification questions in contexts isolated from the draft stops a model from rationalizing its own errors. Both use only the model's internal memory, with no retrieval, and neither studies evaluator bias.

Our claim is specific and testable. When the verifier sees only a canonicalized claim and its evidence, verdict shifts caused by label-preserving cues should fall by at least half compared with monolithic and sentence-decomposed judges, with no loss in hallucination detection. The design also turns bias into something we can locate. If a verdict changes under a cue, either the planner produced a different claim set or the verifier gave a different verdict on the same claim. These two cases can be counted separately. This matters because the judges' own explanations have been shown to hide their shortcuts.

Research Program

The system has three parts.

The planner receives the response one sentence at a time, never the whole answer and never any metadata. It rewrites each sentence into minimal declarative claims. These claims keep content that bears on truth: hedges and modality (so the overconfidence failure of turning tentative evidence into certainty stays checkable), explicit attributions, dates, and quantities. They drop formatting, discourse connectives, and intensifiers. A deterministic coverage check then confirms that every content word of the response is assigned to some claim. Any span left uncovered is marked "unverified" rather than silently dropped. The coverage check exists because a planner that sees fluent text might extract fewer checkable claims from it. That would be a quieter route for bias, and the check makes it visible.

The verifier receives one claim and the retrieved chunks. Adapting the finding of RLSeek that judge errors cluster in reasoning steps that do not quote the source, the verifier must quote supporting or contradicting text before returning supported, contradicted, or baseless. We do not use RL training. Instead, a string match rejects any quote that is not verbatim and forces a retry.

The aggregator is a fixed rule. One contradicted or baseless claim flags the response, and flagged spans are returned for span-level scoring.

One consequence of this design deserves emphasis. An authority cue written into the answer itself, such as "according to a 2024 Nature study," is not removed as noise. The planner turns it into a claim, and that claim is unsupported by the retrieved context. Under the faithfulness definition in RAGTruth, which found that baseless additions outnumber direct contradictions, such an addition is itself a hallucination. A judge that becomes more lenient when this cue appears is therefore wrong twice over.

The analysis centers on a cue-injection suite built on RAGTruth and RAGTruth-Enhance. We apply edits in two classes.

Label-preserving edits should not change the correct verdict:

  • verbosity expansion that only restates supported content;
  • heavy markdown formatting;
  • confident versus neutral tone on supported sentences;
  • provenance and recency labels in the prompt header, following The Silent Judge;
  • moving an existing hallucinated sentence earlier or later, to probe the "tail effect" RAGTruth reports, in which hallucinations cluster toward the end of responses.

Label-changing edits should flip a faithful response to hallucinated:

  • inserting unsupported authority attributions;
  • inserting unsupported dates.

The main bias metric is the Verdict Shift Rate: the fraction of items whose verdict changes under an edit, as used in The Silent Judge. For the blind verifier, each shift is attributed to either the planner (the claim set changed) or the verifier (the same claim received a different verdict).

A secondary study asks whether blindness also helps with the two other known failures of judges. The first is parametric interference, where the judge's own memory overrides the evidence; Bi'an reports that larger judges are worse on counterfactual contexts for this reason. We test this on the counterfactual subset of Bi'anBench. The second is distracting retrieved text: evidence that is on-topic but contains subtly altered facts. We test this with Magic Mushroom-style injections into the judge's evidence. We expect weaker gains here, because blindness hides the answer from the verifier but not the corrupted evidence. Reporting this honestly marks the limit of the idea.

Baselines

The naive reference point is a monolithic prompted judge. The same backbone receives the full context and response in one prompt with chain-of-thought and returns a verdict and spans. This is the setting in which RAGTruth reports GPT-4 with optimized prompting at 63.4% response-level F1 and 28.3% span-level F1. It is the reference against which bias reductions are read.

The strongest detection baseline is a fine-tuned detector trained on RAGTruth's training split, following RAGTruth. There, a fine-tuned Llama-2-13B reached 78.7% response-level F1 and 52.7% span-level F1. We retrain it on a modern backbone of matched size, and we also retrain a Bi'an-style SFT judge with structured reasoning. The blind verifier is not fine-tuned. So if it fails to match these detectors on clean F1, that is a real cost to report, not something to hide. On ME-FEVER we compare against Halu-J's reported 91% accuracy, where GPT-4o scored 83%. On Bi'anBench the only reported figure is Bi'an-14B's 90.1% on the Chinese subset, against GPT-4o's 90.7%. We found no reported result on Lacuna for the counterfactual subset, so we will measure it ourselves.

The closest structural baseline is RT4CHART, reimplemented with its local and global verification. Lacuna reports its cost as about $0.0104 per sample with GPT-4o-mini, but we found no reported result on Lacuna for its RAGTruth-Enhance detection F1 in a form we can quote as a target.

The key ablations isolate blindness itself. Each takes our pipeline and restores exactly one channel to the verifier:

  • the full response;
  • the header metadata;
  • the original, non-canonicalized sentence in place of the canonical claim.

A fourth ablation replaces the fixed aggregator with an LLM aggregator. This tests whether bias re-enters at the final step.

All systems use the same backbones, Qwen2.5-14B and GPT-4o-mini, the same retrieved context, and the same prompt-tuning budget, fixed at an equal number of development-set iterations on a held-out split of RAGTruth. We report total tokens per item because factored verification costs more calls.

The blind verifier beats a baseline if two things hold. Its mean Verdict Shift Rate on label-preserving cues must be at most half the baseline's. Its span-level F1 on RAGTruth-Enhance must stay within two points of the best prompted baseline on the same backbone.

Why not just…?

Why not just strip the cues or paraphrase the answer into neutral style, then judge monolithically? This is the most serious threat, so we run it as a baseline: a rewriter normalizes style, deletes metadata, and a monolithic judge scores the result. Metadata deletion will trivially remove header provenance and recency bias, which is why header cues are not our headline result. Two cases separate the approaches. First, a neutral rewrite keeps authority attributions written into the answer, because they are content. Second, the judge still sees the whole answer, so a confident, coherent body of text can pull a single weak claim toward "supported." This is the local-coherence effect that Local Coherence or Global Validity? documents, where GPT-4o tagged subtle errors in fluent traces with only 57.8% accuracy. The verifier-restored-full-response ablation tests this directly. If paraphrase-then-judge matches the blind verifier on embedded-attribution and style cues, we will conclude that normalization, not decoupling, does the work, and we will report the simpler method as the recommendation.

Why not just fine-tune the judge on cue-augmented data? A detector trained on perturbed pairs, in the style of Bi'an or Halu-J, may well learn to ignore the cues it was trained on. The question is whether it resists cues it has never seen. We hold out entire cue families, such as training on verbosity and provenance and testing on tone and embedded dates, and compare shift rates on the held-out families. Blindness does not depend on knowing the cue in advance, so it should hold up on unseen families where augmentation degrades. If augmentation generalizes equally well, the case for our architecture rests only on its lack of training cost and its ability to localize bias.

Why not just ensemble several judge prompts and keep consistent verdicts? ConsJudge treats agreement across prompt variants as a proxy for truth and trains on it with Direct Preference Optimization (DPO). A shortcut that fires under every phrasing, as the Silent Judge biases appear to, produces consistent wrong verdicts. We run a matched-compute consistency ensemble of five prompt variants with majority vote. We predict that its shift rates stay close to the monolithic judge's even as its self-consistency rises.

Proposed Experiments

The decisive experiment runs the cue suite on RAGTruth-Enhance across all systems and both backbones. It reports three things per cue family and system: the label-preserving shift rate, detection sensitivity on the label-changing edits, and span-level F1 on unedited items. We test significance with paired bootstrap resampling over items.

The attribution analysis follows. For every shift the blind verifier still shows, we record whether it arose in the planner or the verifier, and how often the coverage check caught dropped spans. If most remaining bias sits in the planner, the next step is a planner that is blind across sentences. If it sits in the verifier, the evidence itself is carrying the bias.

Generalization runs on ME-FEVER and the Bi'anBench counterfactual subset. There we measure whether the evidence-only, quote-checked verifier narrows the counterfactual gap Bi'an attributes to parametric interference.

The noise stress test varies distracting evidence from 0% to 60% in the judge's retrieved context, following Magic Mushroom's protocol. It asks whether blindness to the answer buys any robustness to corrupted evidence. Our prediction is that it buys little. Confirming that would bound the claim to biases that come from the answer, not from the evidence.

The idea is falsified if the paraphrase-then-judge baseline or the verifier-sees-full-response ablation matches the blind verifier's shift reduction. It is also falsified if the verifier's reduction in bias comes with more than a two-point drop in span-level F1.