A common way to make a language model less likely to invent facts is to have a stronger model act as a grader, called an LLM judge. The judge picks the better of two answers, and those picks become training pairs for direct preference optimization (DPO), a fine-tuning method that raises the probability of the preferred answer relative to the rejected one. This works when the judge can tell right from wrong. In expert domains the judge often cannot. GPT-4 detects medical hallucinations at a macro-F1 of 0.56, the same as untrained laypeople and far below physicians at 0.79 (MedHalu). Frontier judges also shift their verdicts on authority and recency labels while never admitting it (The Silent Judge). And LLMs that correctly flag "spin" in cancer trial abstracts still let it inflate their efficacy ratings (Caught in the Web of Words). DPO optimizes against whatever the judge rewards. So a model aligned this way should learn to produce the errors its judge cannot see, stated in the confident, expert-sounding style the judge prefers. If the same kind of judge is then used for evaluation, the problem stays hidden.
We propose to measure this inheritance directly and to fix it cheaply. The fix has two steps. First, before any training, we audit the judge with expert-verified minimal pairs to find which kinds of domain error it reliably catches. Second, we trust its labels only on those kinds. For its blind spots we substitute counterfactual pairs that isolate a single verified error and control for tone. The claim is that most of the damage from judge-labeled DPO in medicine is concentrated in a small, measurable set of error types. A per-type audit that costs a few thousand judge calls is enough to route around it. We also claim the damage is invisible to same-family judge evaluation and visible to expert-rubric evaluation.
Core Claim
Our central quantity is the laundering rate. It is the fraction of policy outputs that the training judge accepts but expert-grounded evaluation rejects. We predict three things. Standard judge-labeled DPO in a medical domain raises the laundering rate even while the judge's own hallucination score improves. The increase clusters in error types where the pre-training audit shows low judge reliability; we expect procedural and temporal errors, which are the categories MHB found most exploitable. It also clusters in outputs with confident, authoritative phrasing. Finally, gating judge labels by audited per-type reliability, and filling the gaps with style-matched counterfactual pairs, lowers expert-graded hallucination more than unaudited judge DPO does, without the utility collapse seen when training against continuous factuality scores.
The novelty lies in three pieces used together. The first is diagnosing a judge's domain blind spots at the level of error type, before training. The second is using that diagnosis to decide which judge labels enter DPO. The third is testing whether the policy inherits exactly those blind spots. The contrast with prior work is specific. Direct Judgement Preference Optimization trains strong judges (SFR-Judge) and uses them as reward signals for policy DPO, but only in general-domain chat, and it never asks whether judge errors are inherited. Filtered DPO filters preference pairs with a reward model it assumes is correct. We ask where that assumption fails and filter on the grader's own reliability instead. Automatically Finding Reward Model Biases discovers biases through counterfactual pairs, including a preference for hallucinated details. We reuse its minimal-pair probing, but with expert-verified clinical errors, and we connect the probe results to training outcomes.
Research Program
The audit is the first component. From the training split of MedHalu's expert-validated question-answer pairs, and from PMC-Patients case reports held out from MHB's test cases, we build minimal pairs. Each pair consists of a correct answer and the same answer with one injected, physician-verified error. The errors are tagged with MHB's five categories (factual discrepancy, cognitive inconsistency, diagnostic reasoning, procedural error, temporal obsoletion) and MedHalu's split between fact-conflicting and context-conflicting errors. For every pair we also create cue variants. In one, the erroneous answer is rewritten in confident, authoritative prose while the correct one hedges. In others, provenance labels ("physician-written" versus "AI-generated") are attached following The Silent Judge, or the answer is phrased with trial-report spin. From the audit, each judge receives a reliability score $r_k$ for each error type $k$: the fraction of pairs of type $k$ it resolves correctly under both cue orderings. It also receives a verdict shift rate for each cue. A second domain gives a check that does not depend on physicians. On TheoremQA (TheoremQA), final answers are verifiable, so the judge's reliability at spotting a misapplied theorem precondition in a reasoning chain can be scored automatically.
The training component starts from on-policy pairs. We sample two responses from the policy for each domain prompt, drawn from HealthBench-style dialogues and consumer health queries disjoint from all evaluation sets. The judge labels each pair and states which error type decides its verdict. Pairs decided on a type with $r_k$ above a threshold are kept and weighted. Pairs decided on low-reliability types are dropped. In their place we add counterfactual pairs in which the rejected response differs from the chosen one only by a verified error of that type, and in which the chosen response is at least as confident in tone as the rejected one. This last constraint is there so the policy cannot learn "hedging wins." The weighted objective is
$$\mathcal{L} = -\mathbb{E}_{(x,y_w,y_l,k)}\left[ r_k \log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right)\right],$$
where $x$ is the prompt, $y_w$ and $y_l$ are the chosen and rejected responses, $k$ is the error type that decides the pair, $\pi_\theta$ is the policy being trained, $\pi_{\text{ref}}$ is the frozen starting model, $\beta$ controls how far the policy may move from $\pi_{\text{ref}}$, and $\sigma$ is the logistic function. Counterfactual pairs receive $r_k = 1$ because their labels are expert-verified.
Because Capability Traps in DPO shows that large $\beta$ can permanently damage reasoning while surface benchmarks still rise, we keep $\beta$ in a low range, sweep a small grid of values, and track that paper's internal capability probes alongside the domain metrics.
The analysis component tests whether the audit predicts inheritance. We fit the per-type change in the laundering rate after unaudited DPO against the pre-training $r_k$. A strong negative relationship means blind spots transfer into the policy. The policy is Qwen3-8B, which matches Binary RAR so that numbers are comparable. We use two training judges: an open SFR-Judge-class 8B model and GPT-4o. This lets us see whether blind spots and their inheritance depend on the judge.
Baselines
The naive reference point is the untuned Qwen3-8B policy, together with plain supervised fine-tuning on the same domain prompts using the judge-preferred responses. These establish how much of any gain comes from preference learning rather than from exposure to the domain.
The most direct competitor is unaudited judge-labeled DPO. It is the pipeline Direct Judgement Preference Optimization uses when it applies SFR-Judge as a reward signal for policy DPO. That pipeline is the natural default practitioners would adopt, and its judge is strong: SFR-Judge-8B reports 80.91% pairwise accuracy against GPT-4o's 76.78% across its benchmark suite. To keep the comparison fair, it receives the same prompts and the same on-policy samples, and the same number of preference pairs after our gating. It also gets the same $\beta$ grid and seed count. We report its hallucination rate both under the training judge and under expert-grounded evaluation.
The strongest filtering competitor is Filtered DPO. It discards pairs whose chosen response scores below the policy's own sample under a reward model. It reports a Gold RM score of 1.94 against 1.42 for standard DPO on Anthropic HH Helpful. We run it with the same judge acting as its reward model, so the only difference is whether filtering uses the judge's opinion or the judge's audited reliability.
The strongest factuality method in the file is Binary RAR (Train for Truth, Keep the Skills). It trains with online GRPO against a retrieval-grounded binary verifier and reports a 39.3% reduction in long-form and a 54.4% reduction in short-form hallucination on Qwen3-8B, with only a -1.4% AlpacaEval change. We run it on our medical prompts, retrieving over PubMed-style literature. We match its budget to ours in total verifier calls and GPU-hours, which disadvantages our offline method on online sampling but is the honest comparison. We found no reported result on Lacuna for any DPO or RL method evaluated on MHB, so no published number exists for the target table. The reference points are MHB's reported hallucination rates for untuned models: 29.1% for Claude-4-Sonnet and 59.0% for Llama-4-Scout.
Two ablations isolate the key ingredient. Expert-pairs-only DPO trains solely on the counterfactual pairs, with no judge labels. Random-gating DPO drops the same number of judge-labeled pairs, but at random rather than by $r_k$. Beating our method means lower expert-graded MHB hallucination than unaudited judge DPO and fDPO, with non-overlapping seed intervals. It also means a lower laundering rate. And AlpacaEval degradation must be no worse than Binary RAR's reported -1.4%. If our method matches Binary RAR at a fraction of its compute, that counts as a partial win. If it falls below Binary RAR, the practical claim fails, although the inheritance finding stands on its own.
Why not just…?
Why not just use a better judge prompt? Rubric-augmented prompting helps but leaves most of the gap. MedHalu's expert-in-the-loop prompts lift GPT-4 by 6.3%, which is still well below the physicians' 0.79 F1. Joint detection-then-interpretation prompting reduces the LLM spin gap from 3.38 to 1.14, which still exceeds the human experts' 0.71. We include rubric-prompted judges as an audited condition. If rubric prompting pushes all $r_k$ above threshold, gating becomes unnecessary, and the audit will show that directly.
Why not just train on expert-validated hallucination pairs, as the multimodal hallucination-targeted DPO work does? This is the real threat. Work such as Hallucination-targeted DPO and V-DPO shows that targeted rejected responses work for visual errors. Expert-verified clinical pairs are scarce, in the low thousands for MedHalu and MHB, and they are not on-policy. That suggests they may not cover the errors the policy actually makes. The expert-pairs-only ablation settles the question. If it matches the hybrid, the conclusion is that judges add nothing in this domain, and we will report that as the result.
Why not just use retrieval-grounded binary verification (Binary RAR)? It is strong, but its own analysis reports only 53% retrieval recall on factual errors and a 16% false-negative rate. Procedural and temporal clinical errors are plausibly the ones least likely to have a "smoking gun" passage in retrieved text. We break down its gains by MHB category. If Binary RAR also fails on the low-$r_k$ categories, the two approaches are complementary. If it succeeds on them, it dominates and our contribution shrinks to the diagnostic.
Proposed Experiments
The decisive evaluation never uses the training judge's family. We report MHB hallucination rate using its physician-written rubrics, scored by a judge from a different family, and we check a stratified sample of 300 outputs with licensed clinicians to estimate how often that evaluation judge is itself wrong. We also report MedHalu detection F1, both as a probe of whether the policy learned to recognize errors and as a measure of the audited judges. From the 30 spun and neutralized oncology abstract pairs, we report the gap between efficacy ratings for spun and neutral versions, with the human 0.71 as the target. For utility, we report AlpacaEval, the capability probes from Capability Traps, and TheoremQA accuracy. On TheoremQA, open models sit below 15% and GPT-4 reaches 51.0% with program-of-thought prompting.
The first decisive check is inheritance. After unaudited judge DPO, the laundering rate should rise and correlate negatively with pre-training $r_k$ across error types. The same models' scores under the training judge should improve at the same time. The second check tests the cue channel. The fraction of hallucinated outputs written in confident, unhedged prose should increase under unaudited DPO and not under the tone-matched counterfactual pairs. The third is TheoremQA replication, where judge reliability on precondition errors is scored without experts; it tests whether the effect generalizes beyond medicine.
If there is no inheritance, meaning the laundering rate stays flat and is unrelated to $r_k$, then judge blind spots do not propagate through DPO at this scale, and the motivation for auditing fails. We would report that outcome plainly.
References
- MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
- MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
- Direct Judgement Preference Optimization
- Filtered Direct Preference Optimization
- Automatically Finding Reward Model Biases
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- TheoremQA: A Theorem-driven Question Answering Dataset
- Capability Traps in DPO
- Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
- V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization