Does Tailoring an Explanation Make It Look Like a Hallucination?

An explanation written for an expert skips steps the expert already knows. An explanation written for a novice spells those steps out and often leans on analogies that are not literally true. Both are correct choices about how to talk to someone. An automated evaluator, however, may read them as defects. Large language models used as judges (LLM-as-a-judge: prompting a strong model to score other models' outputs) are known to reward long, exhaustive answers and to fixate on the wording of the reference answer they are given. Hallucination detectors (tools that flag fluent but false statements) are built to catch claims that lack support. So when a system adapts well to its user, the evaluators we rely on may call the compressed expert version "unsupported" and the analogical novice version "false." In long, friendly novice text they may also miss a real error.

This note proposes to measure that confound directly and then remove it. The central object is a set of matched explanations. Each set contains one piece of verified content rendered at several expertise levels, with and without injected factual errors. With these sets we can ask two separate questions of any judge: does its hallucination verdict change when only the audience changes, and does its ability to catch real errors depend on the audience? We then build a judge that keeps two things apart: what an explanation actually asserts, and what it leaves to the reader. It checks facts only against the first, and it scores fit to the audience separately.

Core Claim

The claim has two parts. First, current LLM judges and hallucination detectors are not invariant to faithful re-rendering of content across expertise levels. They produce more false alarms on correctly compressed expert explanations and on correct analogies. They also produce more misses on errors embedded in verbose novice explanations. Second, most of this confound disappears if the judge works in a particular order. It first sorts each gap between stated claims into one of two cases: a gap the stated audience can bridge from background knowledge, or an unsupported leap. It then reduces each analogy to its literal commitment. Only after that does it verify evidence. What is new is the invariance measurement and the audience-conditioned "what is actually asserted" step.

Fact-checking itself is not new here. We reuse the critique-style verification of Halu-J: Critique-Based Hallucination Judge for that part.

Why the gap is real

Three results from different communities point at this failure without testing it. On the explanation side, Hierarchical Expertise Level Modeling for User Specific Contrastive Explanations models a user's knowledge as a level of abstraction and computes the minimal set of facts to state; its explanations often need only one or two predicates. Adaptive Explanations via Direct Preference Optimization trains an explainer to trade faithfulness against how well a simulated doctor or patient can recover the diagnosis. Both show that good adaptation omits material. These papers motivate the work and also supply ground truth, but neither involves a third-party evaluator.

On the judge side, three papers document the relevant biases. Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks finds that GPT-4 agrees with lay users 80% of the time but with dietitians and psychologists only 64% and 60%. It also finds that the judge treats length as clarity and misses clinical safety errors. UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models reports that concise mid-sized models beat verbose general models under expert-validated judging. Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges shows that reference answers induce format bias and "solution fixation," meaning a valid answer is penalized for taking a different path. That solution fixation exceeds 45% attack success for QwQ-32b, and reasoning models are more vulnerable than non-reasoning ones. These papers constrain the design: we should expect reference-conditioned and reasoning-model judges to be the worst offenders, not the fix.

On the hallucination side, HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models shows ChatGPT detects summarization hallucinations only 58.53% of the time. It also shows that side-by-side contrast hurts detection. Halu-J reaches 91% on noisy-evidence fact verification. None of these benchmarks varies the audience.

Research Program

The first component is a matched-rendering benchmark with three sources of ground truth, chosen so that "benign omission" is verifiable rather than asserted.

The first source is symbolic. We take the five International Planning Competition domains used by the expertise-lattice paper. The lattice produces explanations at different abstraction levels that are provably sufficient for the corresponding user model. We verbalize each one with an LLM and check the verbalization back against the planning model. Omitted predicates are therefore known to be benign for that level. Hallucinations are injected as invalid action transitions.

The second source is clinical. We use the CheXpert lung-opacity concept vocabulary from the adaptive-explanation paper, 12 clinical findings mapped to 20 lay terms. We expand it into free-text doctor-level and patient-level explanations, anchored to the same finding set.

The third source is open-domain and financial. We use evidence-grounded items from ME-FEVER and HaluEval QA and summarization, plus UCFE tasks with its four personas. For each item, a generator produces expert, intermediate, and novice renderings from a fixed set of atomic facts. A separate verifier plus human spot-checks confirms that each rendering entails only facts in the set, or analogies whose literal commitment is in the set.

We then inject errors into copies of each rendering, drawing on four hallucination types reported in the literature: entity or role swaps, conditional findings stated as certain, fabricated references, and flawed causal steps. Injections are placed at matched positions across renderings, so the only thing that varies is the surrounding granularity.

The second component is the proposed judge, which works in four stages.

In the first stage, it rewrites the explanation as a list of asserted atomic claims.

In the second stage, it examines each inferential gap between consecutive claims and labels it "bridgeable by the stated audience" or "unsupported leap." To make this label, it asks whether the bridging fact is something a reader at the stated level is assumed to know. This question is posed against an explicit short description of that level's assumed background, which is what distinguishes the stage from free-form "is this complete?" prompting.

In the third stage, it rewrites each analogy as its minimal literal commitment. For example, "the heart is a pump" becomes the claim that the heart moves blood by pressure.

In the fourth stage, it passes only the asserted claims and literalized commitments to a Halu-J-style evidence-by-evidence verifier.

A separate audience-fit score comes from a listener check in the spirit of the adaptive-explanation paper: a model prompted at the target level must answer comprehension questions from the explanation alone. We build a prompted version on GPT-4o and a Mistral-7B version trained with direct preference optimization on the benchmark's matched pairs. The trained version is directly comparable to Halu-J at 7B.

Baselines

The naive reference point is a GPT-4o judge given the explanation and the gold reference text, with no audience information, asked for a hallucination label. This is how most pipelines are run today, and it is where we expect the largest audience-dependent false-alarm gap.

The strongest general-purpose competitor is the same judge with an explicit audience persona and a rubric that tells it not to reward length. The personalized-judge study Can LLM be a Personalized Judge? shows that filtering by verbal confidence above 80 raises agreement above 80% and beats crowdworkers on OpinionQA, 79.2% versus 71.4%. We therefore also run this baseline with that confidence gate and report coverage.

The strongest existing hallucination judge is Halu-J. It reports 91% accuracy on ME-FEVER against 83% for GPT-4o and 78% for base Mistral-7B. No reported result was found on Lacuna for Halu-J or any judge on audience-varied explanations.

We also include reasoning-model judges, DeepSeek-R1 and o1-mini, given the reference. The Curse of Knowledge paper predicts these judges fixate on the reference's structure.

All judges see identical items and evidence. Prompted baselines each get the same budget of prompt-tuning iterations on a shared development split. The trained 7B variant uses the same base model, data scale, and preference-optimization recipe as Halu-J.

We count the proposal as beating the baselines only if three conditions hold together. The false-alarm gap between expert and verbose renderings must fall to within noise, estimated by bootstrapping over items. Detection of injected errors in novice renderings must match detection in expert renderings. Accuracy on the original ME-FEVER must stay within a few points of Halu-J's 91%.

Three ablations isolate the key ingredient. The first removes the gap-labeling stage. The second removes analogy literalization. The third gives the judge a deliberately wrong audience label. If performance is unchanged when the audience label is wrong, the judge is not actually using audience information, and the claimed mechanism is false.

Why not just…?

Why not just prompt the judge to act as an expert? Szymanski and colleagues report that an expert persona raised agreement with specialists by only 4%. A persona also does nothing to separate omission from error: it tells the judge who to be, not what the text leaves to the reader. We include the persona judge as a baseline. If it closes the invariance gap on our benchmark, the proposed pipeline is unnecessary, and we will report that.

Why not just use a claim-level fact checker like Halu-J and ignore the audience? This is the most serious threat. A checker that only verifies stated claims should, in principle, never punish omission. Our first experiment tests this directly by measuring Halu-J's false-alarm rate on correct expert renderings. If that rate is flat across audiences, the omission half of the confound does not exist for claim-level checkers, and the contribution narrows to two things. The first is analogy handling, since a literal checker may label "the heart is a pump" as unsupported. The second is error masking in long novice text, since Halu-J was only tested on short claims. We will say so plainly.

Why not just control for length? Length normalization or a no-verbosity rubric addresses the preference for walls of text. It does not address a correct analogy judged false, or a missing step judged fabricated. The length-controlled judge is a baseline, and the analogy and planning subsets are where it should fail.

Proposed Experiments and Decisive Checks

The first decisive check is whether the confound exists. For each baseline, we compute the false-alarm rate on error-free items for each audience level, and the detection rate on injected items for each level. The planning subset is the clean test, because benign omissions there are provably sufficient. If no baseline shows a significant audience effect, the paper becomes a negative result about a feared failure mode, which is still useful to report.

The second check is whether the proposed judge removes the audience effect without losing sensitivity. We test this on the matched benchmark and on unmodified ME-FEVER and HaluEval, reporting accuracy alongside Halu-J's 91% and ChatGPT's HaluEval figures.

The third check is human validity. Following Szymanski and colleagues' design, domain experts and lay readers label a subset of clinical and financial items. We compare judge agreement with experts against the 73% agreement experts reach with one another. This tests whether audience-invariant verdicts move judge agreement toward specialists rather than lay users.

The fourth check tests whether the reasoning-model paradox transfers. It asks whether reasoning judges show a larger audience gap than non-reasoning judges when given a reference. If so, the practical guidance is that reference-conditioned reasoning judges should not be used to certify adaptive explanation systems.