Explainable-AI systems increasingly hand a model's feature attributions, the numbers saying how much each input pushed a prediction up or down, to a large language model (LLM), which rewrites them as a readable explanation. A second LLM then often checks whether that prose is faithful to the numbers, a practice called "LLM-as-a-judge." We argue that this check has a blind spot that looks like benchmark contamination but is not about leaked test sets. The judge comes to the task already believing things about the features: that education raises income, that "excellent" signals positive sentiment. When an explanation agrees with those beliefs, the judge may accept it even when it contradicts the attribution vector in front of it. When an explanation faithfully reports a model that relies on something surprising, the judge may reject it. We call this prior override. It is a form of contamination in which what the judge absorbed in pretraining replaces the evidence it was asked to check.
The proposal builds a procedurally generated testbed in which the true attributions are known exactly and feature names can be swapped at will. The same mathematical explanation task can then be posed with names that agree with common sense, names that contradict it, or abstract names like $x_3$. Comparing these conditions measures prior override directly. A factorial design separates it from the kinship bias documented in Preference Leakage, and we test a judging procedure that splits explanations into atomic claims and checks most of them by code. The practical stake is concrete. Systems such as iLLuMinaTE report that GPT-4o agrees with human experts 94.68% of the time when auditing explanation fidelity. That kind of number is what licenses deployment, and it was measured on real student data, where the model's behavior and common sense mostly agree.
Core Claim
LLM judges of verbalized attributions are most reliable exactly where they are least needed, when the predictive model behaves the way the world is expected to behave. We predict two failure patterns. First, the judge's false-acceptance rate on unfaithful explanations will be substantially higher when the explanation's claims match pretraining priors than when features carry abstract names. Second, its false-rejection rate on faithful explanations will be substantially higher when the model's true behavior contradicts those priors. We expect both gaps to persist when the judge and the explanation writer come from different model families. That persistence would show prior override is a separate effect from family-style favoritism.
The central quantity is the prior-override gap:
$$\Delta_{\text{acc}} = \text{FAR}_{\text{congruent}} - \text{FAR}_{\text{abstract}}$$
Here $\text{FAR}$ is the fraction of planted-unfaithful explanations the judge rates faithful. The subscript gives the naming condition, holding the attribution vector and the planted error fixed. A symmetric false-rejection gap $\Delta_{\text{rej}}$ is defined on faithful explanations under incongruent naming.
Positioning Against Prior Work
Three papers motivate this work by showing judges can be fooled. Preference Leakage shows that judges inflate win rates for students trained on related models' outputs, with a Preference Leakage Score as high as 37.1% on Arena-Hard, driven by style. That study covers chatbot responses, not attribution fidelity. Its mechanism is also kinship, whereas ours is world knowledge shared by every judge family. FaithCoT-Bench finds that fluent chain-of-thought rationales hide unfaithfulness from judges. Its ground truth is expert annotation of the model's own reasoning, so it cannot vary what the judge believes while holding the facts fixed. TSQueryBench shows judges rank time-series explanations far better than they generate them (Qwen-3 8B: 0.84 ranking accuracy versus 0.38 generation accuracy). Its 500 fixed instances describe signal shapes with no semantic priors, the one setting where our effect cannot appear.
Two lines of work supply the method. DyVal generates reasoning problems from random directed acyclic graphs, so items cannot be memorized and complexity can be controlled. We reuse that idea to generate data-generating processes whose exact attributions are known. M4 shows how to plant synthetic trigger words so a real model's ground-truth reliance is known. We use this for our real-model transfer test.
DCR constrains the design. It probes the evaluated model for four levels of contamination and discounts its accuracy, but it never probes the judge. Our naming manipulation is effectively a contamination probe pointed at the judge. EvalxNLP provides the verbalization pipeline: Llama 3.3-70B over SHAP and Integrated Gradients outputs on MovieReviews, HateXplain, and e-SNLI. It also shows SHAP attributions overlap human rationales only weakly (Token F1 0.316). That weak overlap is exactly the divergence an explanation can exploit by echoing human-style rationales instead of the model's real attributions.
Research Program
The testbed generator samples a random graph over 6 to 20 tabular features. It assigns functional forms (linear terms, pairwise interactions, thresholds) with random signs and magnitudes. It then either uses this function as the predictive model or trains a gradient-boosted model on samples from it. For the closed-form model, exact Shapley values (each feature's average contribution across feature subsets) are computed analytically. For the trained model, they are computed with SHAP and checked against the closed form.
Feature names are drawn from domain schemas for credit, health, education, and sentiment lexicons, under three conditions. Under congruent naming, each name's commonsense direction matches the sampled sign. Under incongruent naming, it contradicts the sign. Under abstract naming, features are called $x_1 \dots x_n$. Because graphs and name assignments are drawn fresh for every evaluation run, no instance can have been memorized.
Explanation writers from four families (Llama, Qwen, Gemma, GPT) verbalize each attribution vector. We then plant controlled errors: sign flips, swapped importance ranks, an omitted top feature, a feature that does not exist, and an unsupported causal or domain claim (for example, "because lenders distrust young applicants"). The first four error types can be verified by a deterministic checker. The fifth is labeled by construction.
Judges are the same four families, given the attribution vector and a fidelity rubric in the style of TSQueryBench. Writer and judge are crossed so the design includes identity, same-family, and unrelated pairings. That crossing yields kinship and naming as two factors whose interaction we estimate.
We then compare three judging procedures. The first is the standard holistic judge. The second is the same judge with feature names replaced by abstract tokens before judging. The third is a claim-split procedure. An LLM first rewrites the explanation as atomic claims of the form "feature, direction, rank." Those claims are checked against the vector by code. The judge sees only the residual claims that do not fit the schema, such as causal and domain assertions, and is asked whether the attribution evidence supports them.
Baselines
The naive reference point is a judge that rates every explanation faithful, together with ROUGE-L against a template verbalization of the true vector. Any useful auditor must beat both on balanced accuracy. On the planted errors these baselines should sit near chance, which would show that the task requires checking against numbers.
The strongest existing method is the holistic attribution-conditioned judge as used in iLLuMinaTE and TSQueryBench. Its reported anchors are GPT-4o's 94.68% agreement with experts in iLLuMinaTE and Qwen-3 8B's 0.84 ranking accuracy in TSQueryBench. Neither paper reports results on prior-incongruent attributions, and we found no such result on Lacuna. We run it with the same rubric, the same inputs, and the same prompt-tuning budget as the claim-split judge: an equal number of prompt variants, selected on a held-out validation set of generated graphs. Our first test is whether it reproduces roughly 90%+ agreement under congruent naming. A cell-by-cell breakdown of accuracy will then show whether that figure hides a large $\Delta$.
The strongest bias-specific baseline is contextual calibration from Preference Leakage, which reduced Error Bias from 17.8 to 7.3. We calibrate it on the same held-out labeled pairs available to our method.
Two ablations isolate the key ingredient. The first, deterministic checker alone, tells us whether the LLM judging the residual claims adds anything. The second, name anonymization alone, tells us whether simply removing priors is enough.
The claim-split judge counts as winning if three things hold. Both $\Delta_{\text{acc}}$ and $\Delta_{\text{rej}}$ must fall within a few points of zero, with confidence intervals from bootstrapping over generated graphs. Its detection of fabricated domain claims must not fall below the holistic judge's. And its overall balanced accuracy must exceed that of calibrated holistic judging.
Why not just…?
Why not just pick a judge from a different model family? That addresses kinship, the mechanism in Preference Leakage. We predict it leaves prior override intact, because every family is pretrained on overlapping web text and so shares the same beliefs about income and sentiment words. The crossed design tests this directly. If $\Delta$ disappears in unrelated writer–judge pairs, our claim collapses into a known effect, and we will report it as such.
Why not just anonymize feature names before judging? This is a real threat, and we treat it as a main ablation, not a straw man. We expect it to eliminate $\Delta$ almost entirely. The cost is that a judge who cannot see names cannot tell whether "because lenders distrust young applicants" is an unsupported domain claim. The fabricated-claim error class exists to measure that loss. If anonymization loses little on that class, it is the recommendation, and the claim-split procedure is unnecessary.
Why not just extract claims and check them by code, with no judge at all? For schema-shaped claims about direction and rank, this should match or beat any LLM, and we expect it to. Explanations in deployed systems like iLLuMinaTE, however, are mostly advice and causal narrative, where fixed rules have historically failed to scale. Where pure checking falls short is an empirical question the checker-only ablation answers. We will report what fraction of real verbalized explanation content even fits the schema.
Why not just use logit- or probability-based detectors? On FINE-CoT, these scored below 50 F1, sometimes under 20. Meanwhile LLM judges reached 65–77 (FaithCoT-Bench). We therefore include them only as a low reference.
Proposed Experiments
The main experiment covers 3 naming conditions, 4 writer families, 4 judge families, and 6 error conditions (5 planted errors plus a faithful control). Each cell gets several hundred freshly generated graphs, sized so a 5-point difference in false-acceptance rate is detectable. We report $\Delta_{\text{acc}}$, $\Delta_{\text{rej}}$, kinship effects, and their interaction with mixed-effects regressions over graphs. A second axis varies graph size and interaction order, as in DyVal's difficulty levels, to test whether prior override grows as the attributions become harder to read.
Two transfer tests move from synthetic to real models. In the first, following M4, we plant a meaningless trigger token in MovieReviews training data so the classifier provably relies on it. Writers then produce one explanation citing the trigger, which is faithful, and one citing sentiment words, which is prior-congruent but unfaithful. In the second, on e-SNLI and HateXplain with the EvalxNLP classifier and verbalizer, we select instances where SHAP and the human rationale disagree. We then ask judges, given SHAP values, to choose between an explanation faithful to SHAP and one paraphrasing the human rationale. Any preference for the rationale-style explanation is direct evidence that memorized human rationales override evidence.
The decisive check is simple. If $\Delta_{\text{acc}}$ stays below roughly 3 points across judges and the trigger test shows no preference for the sentiment-word explanation, then reported audit accuracies like iLLuMinaTE's generalize to counterintuitive models, and our concern is retired. If the gaps are large, persist across unrelated families, and are removed by claim-split judging without losing fabricated-claim detection, then current practice for auditing verbalized explanations needs to change. The change would be to report fidelity separately on prior-incongruent cases and to route schema-shaped claims through code.
References
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- iLLuMinaTE: An LLM-XAI Framework Leveraging Social Science Explanation Theories Towards Actionable Student Performance Feedback
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
- TSQueryBench: LLM-as-a-Judge for Time-Series Explanations
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models
- DCR: Quantifying Data Contamination in LLMs Evaluation
- EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models