Judging What a Claim Implies, Not Just What It Says

Much misinformation is not false sentence by sentence. A post can state two accurate statistics side by side so that readers infer a causal link that does not exist. It can quote a real study while leaving out the caveat that reverses its meaning. Professional fact-checkers have labels for this, such as "Mostly True," "Partly True," or "Missing Context." Automated pipelines mostly do not. The dominant automated method, decompose-then-verify, splits a claim into small standalone facts (atomic sub-claims), checks each one against evidence, and calls the claim false if any piece fails. A claim built entirely from true pieces therefore passes, however misleading the whole is.

We propose an LLM judge (a large language model prompted to grade a claim against evidence) that grades a claim on two separate axes. The first is the literal accuracy of what it says. The second is the accuracy of what a typical reader would conclude from it. The judge first writes down that implied conclusion explicitly. It then retrieves evidence for the conclusion itself, not only for the literal atoms, and maps the pair of verdicts onto a graded veracity scale. The idea in two sentences: misleading-but-true claims are exactly the cases where literal verification and implied-conclusion verification disagree. A judge forced to check both should separate "True" from "Mostly True" and "Partly True" far better than one that checks atoms or reads the claim as a whole.

Core Claim

The gap is specific. FactLens decomposes compound claims and shows that sub-claim verification holds roughly 80% F1 on complex claims, where holistic verification falls to roughly 55–60%. Its aggregation is conjunctive, though, so it has no slot for a claim whose parts are all true. Decomposition Dilemmas catalogues how decomposition damages meaning, through omission of context, ambiguity, over-decomposition and alteration of meaning. It also shows that decomposition hurts on short single-sentence claims. That second finding constrains us: we cannot simply decompose harder.

FactBench's VERIFY pipeline labels units as Supported, Unsupported or Undecidable. Verifying the Verifiers sets aside its ambiguous GRAYFACTS split to build a clean binary benchmark. Neither grades the space where misleading claims live. DiVA produces a continuous factuality score from a learned scoring layer, but that score measures how much of an answer is correct, not whether a correct answer misleads, and it gives no reasons.

Our claim is twofold. First, explicitly stating and verifying the reader's implied conclusion is the ingredient these systems lack. Second, its benefit should be concentrated on the intermediate labels (Mostly-True, Partly-True, Mostly-False) and on claims whose atoms are all true. If the gain is uniform across labels, or disappears once a holistic judge is simply told to look for missing context, the claim is wrong.

Research Program

The judge runs four steps on each claim and its evidence.

  1. Implication step. It writes the conclusion a typical reader would draw. It may write two or three if the claim is genuinely open to more than one reading, and it records which phrases trigger each one.
  2. Literal check. It verifies the literal content, splitting the claim only when it is compound. A simple claim-complexity test decides whether to split, which addresses the harm to simple claims reported in Decomposition Dilemmas.
  3. Implied check. It issues fresh retrieval queries aimed at the implied conclusion and verifies that conclusion separately.
  4. Grading. It combines the two verdicts into a grade. Literal-true with implied-true maps to True. Literal-true with implied-false maps to the misleading region. Literal-false maps to the false region. The model refines the grade within each region using the evidence it found.

The rationale the judge writes is built from these intermediate outputs, so every sentence of the explanation points to a retrieved passage.

Two lines of work supply method rather than motivation. Self-rationalization shows that having an 8B judge reason before it scores, then aligning it with preference optimization, improves fine-grained Likert scoring by 3–9% over plain fine-tuning. We reuse that recipe for our open-weight variant, with the implication and the two verdicts serving as the structured reasoning. Trust or Escalate shows how to send low-confidence cases from a cheap judge to an expensive one with a statistical guarantee on agreement with humans. We use it only to control cost, escalating claims where the cheap judge's implication step is unstable across samples.

Two findings from the file constrain the design.

  • Unfaithful explanations. MisinfoEval found that GPT-4 produced erroneous justifications in about 24% of cases even when given the correct label. We therefore treat the implication step itself as a possible source of error. We measure it directly against human-written reader conclusions rather than assuming it is right.
  • Hidden shortcuts. The Silent Judge shows that judges shift verdicts on provenance and recency cues, for example +18% for "EXPERT" tags on GPT-4o, while acknowledging those cues 0% of the time. Fact-check claims arrive with claimants and outlets attached, so we audit whether our judge leans on such cues more in the ambiguous middle of the scale, where literal evidence underdetermines the grade.

The data we build has two parts.

  • Reader-conclusion annotations. For about 1,000 claims drawn from the English and two non-English portions of X-Fact, crowd readers write what they would conclude from each claim. Separate annotators then mark whether that conclusion is supported by the fact-checker's article.
  • Controlled true-but-misleading set. Starting from evidence documents in CLEARFACTS, annotators compose claims that use only verified sentences but select or juxtapose them to imply a false conclusion. Each is paired with a faithful control built from the same sentences.

The controlled set isolates the phenomenon. The X-Fact annotations test whether it matters on real fact-checks.

Baselines

Naive reference: majority class plus claim-only. The do-nothing reference is the majority-class predictor together with the claim-only classifier from X-Fact. That paper reports 38.2% in-domain macro-F1 for claim-only, against 41.9% for its attention-based evidence aggregator with metadata, and roughly 15–16% macro-F1 for both on out-of-domain sources and unseen languages. We rerun both on the paper's in-domain, out-of-domain and zero-shot-language splits. Any LLM judge must clear 41.9% in-domain to be interesting. It must clearly exceed 15–16% out of domain to show it is verifying claims rather than memorizing sources.

Strongest existing method: few-shot frontier holistic judge. The strongest existing competitor is a frontier model given the full claim and evidence, with a few-shot prompt containing the seven X-Fact labels and their definitions. This is the setup that Verifying the Verifiers found takes o1 from 9.4 to 88.7 macro-F1 on binary CLEARFACTS. That number is binary and on a different dataset. I found no reported result on Lacuna for an LLM judge on X-Fact's seven-class task, so we establish it ourselves. The baseline gets the same model, the same retrieved evidence, and the same number of few-shot examples selected on the same development split as our judge. Its prompt-tuning budget equals ours, measured in development-set evaluations.

Strongest decomposition method: graded FactLens pipeline. We implement the FactLens decompose-then-verify pipeline with a graded aggregation instead of the conjunctive one. The fraction of atoms that are false maps to the intermediate labels, with thresholds tuned on development data. This gives decomposition its best fair shot at nuance.

Key ablation. The ablation that isolates our ingredient removes the implication step and implied-conclusion retrieval and keeps everything else identical.

What counts as winning. Beating these baselines means a statistically significant gain in macro-F1 over the intermediate labels on X-Fact, and in recall on the controlled misleading set, at matched model and retrieval budget. The ablation must lose most of that gain.

Why not just…?

Why not just tell a strong holistic judge to check for missing context? This is the most serious threat. A frontier model reading the whole claim may already infer the implied conclusion internally. We include exactly this variant: the holistic baseline with an added rubric line about omission and framing. If it matches our judge on the controlled misleading set, our explicit step adds nothing beyond prompting, and we will report that. The distinguishing prediction is that the holistic judge retrieves and attends to evidence for the literal claim. It should therefore miss misleading claims whose counter-evidence appears only under queries about the implied conclusion. We test this by comparing performance when retrieval is shared versus when each system issues its own queries.

Why not just grade decomposition by the fraction of false atoms? On truly misleading claims every atom is true by construction, so fraction-based grading must output True. The controlled set measures this directly: graded FactLens should score near zero recall on the misleading half while matching our judge on the faithful controls. On real X-Fact data the picture may be less clean, because many Partly-True claims do contain a false atom. If graded decomposition matches us there, our contribution narrows to the omission subtype, and we will say so.

Why not just train a classifier on X-Fact's labels? X-Fact already shows that evidence-free and evidence-aggregating classifiers fall to about 15–16% macro-F1 out of domain. Their in-domain scores are largely explained by claim style and source. We run the classifier on the out-of-domain and zero-shot-language splits. A judge that reasons about evidence should degrade far less.

Why not just output a DiVA-style scalar score? DiVA reports 84.24% Precision@1 on Natural Questions versus 77.64% for FactScore. However, it scores answer correctness, and a scalar cannot say that a claim is literally true but misleading. MisinfoEval found reasoning-based explanations improved user accuracy by up to 41.72%, against 24.2% for labels alone. We do not claim to beat DiVA on its benchmark, and we do not run it.

Proposed Experiments

Experiment 1: controlled misleading set. We report recall on misleading claims and false-alarm rate on matched faithful controls for every system. This is the cleanest test of the mechanism.

Experiment 2: X-Fact. We report seven-class macro-F1 on all three splits and break it down by label. The decisive quantity is the confusion between True and the Mostly-True/Partly-True labels. Our hypothesis predicts the gain over the holistic baseline concentrates there and on claims whose atoms we independently verify as all true.

Experiment 3: quality of the implication step. We measure agreement between the judge's stated implied conclusions and the crowd-written ones, using human-rated semantic match. We also measure how often a wrong implication causes a wrong grade. If implication errors approach the 24% explanation-error rate from MisinfoEval, the method is limited by that step, and we report it as the main failure mode.

Experiment 4: GRAYFACTS. On an annotated subset of GRAYFACTS, which has no reported numbers on Lacuna, we test whether the two-axis grade separates "misleading" from "genuinely unverifiable." Today these are lumped together as ambiguous.

Experiment 5: provenance audit. Following The Silent Judge, we inject claimant and outlet cues, such as an official agency versus an anonymous blog, and measure verdict shift rate. We compare intermediate-label claims with clear True/False claims, and our judge with the holistic baseline, plus a variant with the cues stripped. If shortcut reliance spikes in the middle of the scale for all systems, any graded judge must be run without provenance metadata. That is a deployment constraint worth establishing on its own.