Rhetoric That Hides Errors and Rhetoric That Adds Them: Testing LLM Judges on Both

Large language models are now routinely used as automatic graders of other models' outputs, including the question of whether a summary or answer is factually faithful to its source. These graders have a documented weakness for surface cues. This proposal targets a specific and practically common version of that weakness: persuasive wording inside the graded text itself. A summary can be factually perfect but written in charged, authoritative language. It can also be wrong in a way that lives entirely in the rhetoric. A finding the source called tentative becomes "established". Two facts the source merely listed become linked by "because". Those wording choices are factual claims, even though they read like style.

We argue that LLM judges fail on rhetoric in two opposite directions, and that each obvious fix repairs only one of them. Ornamental rhetoric, such as loaded adjectives, flag-waving or a confident expert tone, shifts verdicts without changing what is true. Truth-conditional rhetoric smuggles in unsupported claims that judges treat as framing. Examples are certainty upgrades, inserted causal connectives, and strengthened comparatives or quantifiers. Stripping all style before judging removes the camouflage but also deletes the smuggled claims. Breaking text into atomic facts tends to drop exactly the causal and modal glue where the error sits. The proposal builds a controlled benchmark that separates these two cases at the span level. It then tests a judge that tags rhetorical spans and routes them: ornamental spans are neutralized, and truth-conditional spans are rewritten as explicit claims to verify against the source.

Core Claim

The claim is that rhetoric affects an LLM judge's factuality verdict through two mechanisms that can be separated experimentally. The first is camouflage, where non-truth-conditional persuasion changes verdicts on texts whose factual status is fixed. The second is smuggling, where truth-conditional rhetoric introduces errors that judges detect less reliably than equally severe entity or date swaps. A rhetoric-aware decomposition that treats the two span types differently should beat both style normalization and atomic-claim decomposition on the combined benchmark. Neither of those simpler methods should be able to win on both halves at once.

What would be new is the factorial design itself. Factual status and rhetorical device type are crossed under human verification, so the effect of rhetoric on a judge can be measured rather than inferred. The routing judge is a direct test of whether span-level rhetorical analysis adds anything beyond "rewrite it plainly".

What Prior Work Leaves Open

Two papers motivate the camouflage half. The Silent Judge shows that GPT-4o and Gemini-2.5-Flash flip pairwise verdicts when candidates are tagged with provenance or recency metadata, by up to +30% on ELI5. Their rationales never mention the cue (0% acknowledgment). Its cues are external tags, not language inside the candidate. Do Automatic Factuality Metrics Measure Factuality? shows that appending a content-free sentence like "The summary entails the information the document discusses" raises factuality scores by 0.1 to 0.2. Its manipulation is a single fixed template, not the range of persuasive techniques models actually produce. We extend both from tags and templates to naturally phrased, span-level rhetoric, and we hold factual status fixed by construction.

The smuggling half is motivated by the hallucination literature's own failure reports. GenAudit reports roughly 40% real-world error recall. Its authors state that logical and discourse-level errors remain unsolved, and that their system mostly catches entity and date swaps. SummEdits restricts its edits to atomic substitutions. FAVA builds a span-level hallucination taxonomy (Entity, Relation, Contradictory, Invented, Subjective, Unverifiable). That taxonomy has no category for a certainty upgrade or an inserted causal link. It also notes that over 40% of human-annotated hallucinations are "unverifiable", which is where overconfident framing would land. Halu-J structures evidence-by-evidence critique at the claim level, but does not look at how a claim is phrased.

The method comes from fine-grained propaganda analysis. Da San Martino et al. and SemEval-2020 Task 11 define span identification and technique classification and supply the PTC corpus for training a tagger. Lei et al. show that manipulative content concentrates in sentences carrying Contingency (causal) and Comparison discourse relations. That is direct evidence for our hypothesis that truth-conditional rhetoric lives in discourse connectives. We reuse their discourse-distillation approach to tag these relations.

One constraint comes from the same literature. Token taggers reach about 77 F1 on loaded language but as low as 25 on structural fallacies SemEval-2020 Task 11. We therefore restrict the routing method to devices with clear surface signatures and do not claim to detect straw men.

The Counterfactual LLM-based Framework for Measuring Rhetorical Style is the closest methodological neighbor. It separates rhetoric from content by extracting the substance of an abstract and generating counterfactual rewrites. It produces one document-level scalar and assumes the extracted content is factual. Our design inverts this. Factual status is the controlled variable, and rhetoric is varied at the span level. That paper also reports near-zero correlation between rhetorical hype and peer-review scores, which suggests expert humans resist hype. This is a useful contrast, and we measure human annotators on the same items to test it.

Finally, HaluEval reports that showing a judge a correct and a hallucinated version together degrades detection. We therefore evaluate single responses, not pairs, so that this effect does not confound our results.

Research Program

Building the benchmark (RhetEdits). The benchmark starts from SummEdits seed summaries that humans verified as consistent with their source documents (Cohen's kappa 0.9 in the original annotation). Each seed is edited in one of three ways.

Ornamental edits insert PTC-style techniques that do not change what the summary asserts about the source. These include loaded language, name-calling of non-central parties, flag-waving, authoritative-tone framing ("experts agree that…" only where the source supports the agreement), and the template assertion from the metrics-stress paper as a reference point.

Truth-conditional edits change one phrase so that the summary asserts something the source does not support. These include certainty upgrades ("may reduce" becomes "reduces"), causal insertion (two listed facts joined by "because" or "which led to"), comparative or quantifier strengthening ("some" becomes "most", "similar" becomes "better"), and a temporal relation recast as a causal one.

Control edits are a matched set of entity and date swaps in the SummEdits style, plus meaning-preserving neutral paraphrases. We also apply ornamental edits to inconsistent summaries, giving the full cross of factual status and device type.

Each item records the edited span and its device type. Every item is verified by trained annotators rather than crowdworkers, following the finding of Da San Martino et al. that crowdsourced propaganda labels are unreliable. Items are dropped where annotators disagree about whether the source supports a truth-conditional edit. A smaller second split applies the same edit families to responses in FAVA's FAVABENCH to test transfer to open-ended information-seeking text.

Measuring judge behavior. For each judge and edit family, two quantities are reported. The first is balanced accuracy on the consistency label. The second is a verdict shift rate, adapted from the Silent Judge paper:

$$ \mathrm{VSR}_d = \Pr[\,J(x_d) \neq J(x)\,] - \Pr[\,J(x_{p}) \neq J(x)\,] $$

Here $J$ is the judge's binary verdict, $x$ is the unedited item, $x_d$ is the item after a device-$d$ edit, and $x_p$ is a neutral paraphrase of $x$. Subtracting the paraphrase term removes the instability any rewrite causes. For ornamental devices any positive shift is an error, since the label did not change. For truth-conditional devices the correct behavior is to shift, so we report the miss rate against the matched entity-swap miss rate. We also report how often the judge's rationale mentions the rhetorical span, as an analogue of the acknowledgment rate.

The rhetoric-aware judge. The routing judge has three stages.

The first stage tags spans. It uses a tagger fine-tuned on PTC-SemEval20 for ornamental techniques, plus a discourse-relation and modality tagger distilled in the manner of Lei et al.. The second tagger finds Contingency and Comparison connectives, certainty markers and quantifiers. Neither tagger is trained on RhetEdits, which keeps the benchmark honest.

The second stage routes the tagged spans. Ornamental spans are rewritten neutrally in place. Each truth-conditional span becomes an explicit standalone claim, such as "The source states that A caused B" or "The source presents finding F as established rather than tentative." The remaining text is decomposed into atomic claims as usual.

The third stage verifies every claim against the source with the same base judge used in the baselines, using a Halu-J-style evidence-by-evidence prompt. The summary is consistent only if all claims pass. Because the base judge is held fixed, any gain comes from the routing rather than from a stronger model.

Baselines

The naive reference point is the base judge reading the edited summary with a standard chain-of-thought faithfulness rubric and no defense. We run it with a GPT-4-class model and an open 7B-class model. Majority-class and SummEdits human accuracy (90.9% balanced accuracy, SummEdits) bracket the scale. On the original SummEdits, GPT-4 reaches 82.4% balanced accuracy SummEdits. The first question is how far this falls on each RhetEdits cell.

The strongest existing specialized checkers are three systems. The GenAudit backend reports 74.75 balanced accuracy on SummEdits, and it is the most directly comparable trained auditor on the same seed distribution. Halu-J reports 91% accuracy on ME-FEVER against GPT-4o's 83%. On FAVABENCH, FAVA reports 48.1% detection F1 against GPT-4's 42.5%. All three run with released weights on identical inputs, with no task-specific tuning, matching the budget given to our method. For the NLI-style metrics evaluated in the metrics-stress paper, we found no reported result on Lacuna for RhetEdits-style edits, so they are included as unreported reference points only.

Two simple defenses are the real competitors. The first is style normalization: the same base model rewrites the summary "plainly, preserving every factual claim" before judging. The second is atomic-claim decomposition followed by per-claim verification. Both use the same base model, the same verification prompt, and the same number of model calls as ours, give or take one stage. Each gets the same prompt-tuning budget of a fixed number of variants, selected on a small development split.

Ablations isolate the key ingredient. One version neutralizes all tagged spans without turning truth-conditional spans into claims. Another turns spans into claims but leaves ornamental spans in place. A third uses oracle gold spans, separating tagger error from routing benefit.

Beating the baselines means two things at once. The method must match style normalization on the ornamental cells, meaning a near-zero paraphrase-corrected VSR. It must also significantly beat both normalization and decomposition on truth-conditional recall, using a paired bootstrap over items. Winning only one side does not count.

Why not just…?

Why not just rewrite the text plainly before judging? This is the most serious alternative, and it may fully solve camouflage. Our hypothesis is that a plain rewrite also flattens certainty upgrades and causal connectives, because a rewriter told to remove "style" treats "clearly" and "because" as style. RhetEdits measures this directly as truth-conditional recall under normalization, alongside an audit of how often the rewrite deleted or softened the edited span. If a careful normalization prompt preserves those spans and matches our truth-conditional recall, the routing stage adds nothing. In that case we would report the benchmark and the normalization result as the contribution and drop the method.

Why not just decompose into atomic facts and check each one? Decomposition already removes most ornament. Its known weakness is that relational and logical content is where checkers fail: GenAudit reports about 40% real-world recall concentrated on entity swaps. A decomposer that splits "A happened, which led to B" into "A happened" and "B happened" verifies two true facts and loses the error. The causal-insertion and certainty-upgrade cells test exactly this. If decomposition keeps those links as claims, it is equivalent to our method and the comparison will show it.

Why not just tell the judge to ignore tone and persuasive language? The Silent Judge result argues against it. Judges swayed by cues never acknowledge them, which suggests the influence is not something the model reports and can be told to switch off. An instruction to ignore style may also make the judge discount certainty and causal wording, repeating the normalization failure. We include this as a prompt-only baseline under the same tuning budget.

Why not just fine-tune the judge on rhetorically augmented data? This is a real threat on in-distribution devices: a judge trained on RhetEdits-style edits would likely learn them. The question is whether it generalizes to devices it has not seen. We test this by training on some device families and holding out others, for example training on certainty upgrades and testing on causal insertion. The routing method uses taggers trained only on external propaganda and discourse data. If the fine-tuned judge generalizes across held-out families as well as routing does, fine-tuning is the better practical answer and we will say so.

Proposed Experiments and Decisive Checks

The first check is whether camouflage exists. On ornamental edits of consistent and inconsistent SummEdits items, does the naive judge show a paraphrase-corrected verdict shift reliably above zero, and does its rationale mention the rhetorical span? A null result here would itself be informative, since it would contradict the extrapolation from the metadata and template studies. It would shrink the proposal to the smuggling half.

The second check is whether smuggling exists. On truth-conditional edits, is the naive and specialized judges' miss rate higher than on matched entity and date swaps of the same seed summaries? This compares GenAudit, Halu-J, and GPT-4-class judging.

The third check is the opposite-failure prediction. Style normalization should do well on ornamental cells and poorly on truth-conditional ones. Atomic decomposition should do poorly specifically on causal insertion. The routing judge, with the base model held fixed, should do well on both.

The fourth check is transfer. Repeating the comparison on the FAVABENCH split tests whether the effect holds for open-ended responses, not just summaries. Separately, running the routing judge on unmodified SummEdits and HaluEval summarization (where ChatGPT detection accuracy is 58.53% HaluEval) checks that the method does not hurt performance on ordinary errors.

Finally, trained human annotators judge a subset of ornamental items, to test whether the expert resistance to hype reported by the counterfactual style study holds for factual judgments. That comparison tells us whether camouflage is a weakness specific to LLM judges or a shared one.