Is It the Claim or the Evidence? Routing Fact-Checking Cascades by Evidence Sensitivity

Automated fact-checkers retrieve documents about a claim and then ask a model whether those documents support or refute it. To save money, a common design runs a small model first and hands the claim to a large model only when the small one looks unsure. This design has a hidden flaw. A small model can be unsure for two very different reasons. The claim may be genuinely hard, which a larger model can help with. Or the retrieved evidence may be irrelevant or misleading, which a larger model reading the same bad evidence usually cannot fix, and which it may even make worse. We propose a router that tells these two cases apart. It asks the small model the same question several times with parts of the evidence removed, including once with no evidence at all. If the verdict swings depending on which documents it sees, the problem lies in the evidence, and the system should re-retrieve or fall back to judging the claim alone. If the verdict stays uncertain whatever evidence it sees, the problem lies in the claim, and the system should escalate to the large model.

The motivation comes from a striking finding. On ClimateCheck, giving LLM verifiers retrieved evidence produced lower accuracy than letting them judge the claim alone, and in oracle experiments, giving verifiers the gold evidence raised accuracy by 14–22 percentage points across all models (How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation). Most of the remaining error in these pipelines is an evidence problem, yet every uncertainty-based cascade we know of answers uncertainty by buying a bigger model. A second, practical point concerns training data. The router's training labels come from actually running each remedy on labeled training claims and recording which one fixed the verdict. No teacher model is asked to judge evidence quality, so the router avoids the synthetic-label ceiling that affects teacher-distilled verifiers.

Core Claim

We claim that, for retrieve-then-verify fact-checking, a single scalar uncertainty score is the wrong input for routing. Splitting uncertainty into an evidence-sensitivity part and an evidence-invariant part, both measured by cheap counterfactual passes of the small verifier, supports a three-way decision: accept, repair evidence, or escalate the model. At a fixed budget of large-model calls, this should give higher macro-F1 than routing on logit margins, semantic entropy, or verdict-type-specific thresholds. The gains should be concentrated on noisy, lexically mismatched benchmarks.

The two quantities are simple to state. Let $p(y \mid c, E)$ be the small verifier's distribution over verdicts $y$ for claim $c$ given evidence set $E$. Let $E_1,\dots,E_K$ be perturbed evidence sets: leave-one-document-out subsets, the top-1 document alone, and the empty set $\emptyset$. Define

$$S(c,E) = \frac{1}{K}\sum_{j=1}^{K} \mathrm{TV}\big(p(y\mid c,E),\, p(y\mid c,E_j)\big), \qquad U(c,E) = H\Big(\frac{1}{K}\sum_{j=1}^{K} p(y\mid c,E_j)\Big),$$

where TV is total-variation distance and $H$ is entropy. High $S$ means the verdict depends on which evidence is present, which points to retrieval as the problem. High $U$ with low $S$ means the model is uncertain whatever evidence it sees, which points to claim difficulty. There is a third pattern: confident but opposite verdicts from different evidence subsets. We treat this as genuine conflict and not as noise. This matters because roughly 20% of well-studied claims in SciFact-Open have truly conflicting evidence, and AVeriTeC includes a Conflicting Evidence label. A router that "repairs" genuine conflict would be erasing real signal.

Relation to Prior Work

Three lines of routing work supply the machinery but share the scalar assumption. The first routes whole queries on semantic entropy, computed over sampled answers clustered by meaning (Leveraging Uncertainty Estimation for Efficient LLM Routing). The second gates slow reflection on the margin between verdict tokens (CAMEL). The third learns a monotone confidence ranker from about 3,000 human-labeled calibration pairs (Margin-Adaptive Confidence Ranking for Reliable LLM Judgement). All three have a single remedy for doubt, which is more compute on the same input. We add a remedy that changes the input instead.

The closest design ingredient is from Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence. That work shows confidence means different things depending on what the small model predicts. Abstentions are precise even at low probability, and type-specific thresholds reached 0.8146 precision while routing 19.3% of traffic, against 26.4% for a global threshold. We reuse this lesson by conditioning on the predicted verdict, since Not Enough Info plays the role of abstention. We differ in splitting confidence by its source, not only by output type. That paper also found its 235B fallback had lower recall than the fine-tuned 4B model on some attributes. This constrains our design: escalation should not be assumed to help.

Self-Knowledge Guided Retrieval Augmentation decides whether to retrieve at all, using nearest neighbours over training queries. It also reports that asking a model directly whether it knows the answer works poorly. That negative result is why we measure sensitivity behaviourally and do not prompt for it. Its decision is binary and happens before retrieval. Ours happens after retrieval and chooses among repair, escalation, and acceptance.

Noise-robust verifiers are complementary and serve as baselines, not competitors in kind. ATTUN trains attention to down-weight distractor evidence using usefulness labels (Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers). Ext2Gen trains extraction-before-answering with DPO on GPT-4o-derived preferences (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation). DiVA runs every claim through search, compression, and scoring (DiVA). None of these allocates compute per claim. CLUE explains uncertainty that comes from conflicting evidence pairs, but explicitly leaves uncertainty from retrieval failure unaddressed, which is exactly what $S$ targets.

On the question of where labels come from, Generative Verifiers and EviPath show that students inherit the quality of teacher rationales, and ATTUN names synthetic usefulness labels as an open risk. This motivates our outcome-grounded labels.

Research Program

The pipeline starts with BM25 retrieval of the top $k$ documents and a small verifier, Llama 3.1 8B, matching the zero-shot setup of the cross-benchmark study. For each claim we run $K$ counterfactual passes of the small verifier. With $k=5$, $K$ is about 7 short forward passes that score only the verdict tokens, so the cost is a small multiple of one pass. That is cheaper than semantic entropy, which samples full generations and runs an entailment model.

The router can take four actions. It can accept the verdict. It can repair the evidence, either by re-retrieving with a dense or hybrid retriever and a small-model query rewrite, or, if the claim-only verdict is confident, by falling back to it. It can escalate the same evidence to Llama 3.1 70B. Or it can both repair and escalate. When confident subsets disagree, it outputs Conflicting or NEI.

Training labels come from execution. On the training split of each benchmark, every action is run for every claim. Each claim is then labeled with the cheapest action that produces the gold verdict. A small gradient-boosted classifier is trained on $(S, U, \text{claim-only agreement}, \text{predicted class}, \text{retrieval score})$ and deployed under a budget constraint. The expensive all-actions sweep is paid once at training time. As an analysis, we compare these labels against teacher-generated ones: 70B judgments of whether each document is useful, ATTUN-style. The question is whether a router trained on teacher labels inherits systematic mistakes, for example treating topically related but non-probative documents as useful. This is the mismatch that drives ClimateCheck failures.

Baselines

The naive reference points are the small verifier alone and the large verifier on every claim. Both use the same BM25 evidence, and these fix the two ends of the cost-accuracy curve. The oracle-evidence condition, where the verifier sees gold evidence, gives an upper bound on what repair could achieve. The cross-benchmark study reports this bound as a 14–22 point accuracy gain.

The strongest published systems give reference numbers on these benchmarks. SANCTUARY reached 0.70 macro-F1 on SciFact but 0.31 on ClimateCheck (How Robust). The same paper reports that peak accuracy on ClimateFEVER hovered around 45%, and that TF-IDF logistic regression beat evidence-conditioned LLMs on ClimateCheck. MultiVerS dropped from 72.5 to 52.4 F1 moving to SciFact-Open. On AVeriTeC the reported baselines are 0.23 macro-F1 for the original pipeline and 0.39 for GPT-3.5-turbo (AVeriTeC). These systems are not cascades, so they anchor absolute quality but not cost.

The strongest competitor on the actual question is scalar-uncertainty routing. We run three versions. The first is semantic-entropy routing following Leveraging Uncertainty Estimation for Efficient LLM Routing. We found no reported result on Lacuna for semantic-entropy routing on fact-checking benchmarks; its only reported number in the file is a cost of $3.74 to reach 80% performance on MT-Bench. The second is a verdict-margin gate in the style of CAMEL. The third uses verdict-class-specific thresholds following the cascade-routing paper, since its type-aware policy is the most direct evidence that structured confidence routing beats a single threshold.

For fairness, all routers share the same small and large models, the same retrieved evidence, and the same training split to tune thresholds or fit classifiers, with an equal hyperparameter search budget. We also give the semantic-entropy router the same number of extra small-model calls our method uses. Beating these baselines means higher macro-F1 at matched large-model call rates of 10%, 20%, and 30%, or equal macro-F1 with fewer large-model calls, on at least three of the four benchmarks, with ClimateCheck and SciFact-Open expected to show the largest gap.

Two ablations isolate the key ingredient. The first gives a scalar router the same features but only the escalate action, which tests whether the gain comes from the repair action rather than better features. The second restricts the perturbations to the empty set only, a claim-only versus full-evidence check, which tests whether finer leave-one-out passes add anything beyond that single comparison.

Why not just…?

Why not just route on semantic entropy or a type-aware threshold? This is the real threat. If most small-model errors on these benchmarks come from claim difficulty, scalar routing will match us, because $S$ carries little information and escalation is the correct remedy anyway. The ClimateCheck result argues otherwise: when evidence hurts even large models, escalating on the same evidence cannot recover the loss. The action-restricted ablation and the per-claim oracle analysis below settle this directly. If fewer than about a fifth of recoverable errors are fixed by repair, we will report that scalar routing suffices.

Why not just send everything to the big model, or use a noise-robust verifier like Ext2Gen or ATTUN? The cascade study reports the large model at 19.8x the small model's cost. The cross-benchmark study found evidence-conditioned LLMs of both 8B and 70B losing to claim-only judgment on ClimateCheck, so scale alone does not remove the evidence problem. Robust verifiers reduce sensitivity to distractors but still cannot recover evidence that was never retrieved. We include an Ext2Gen-trained 8B verifier as the small model in one arm. If routing adds nothing on top of it, robustness training has already absorbed the benefit.

Why not just decide retrieve-or-not up front, as SKR does, or ask the model whether it needs more evidence? Deciding before retrieval cannot detect that a specific retrieved set is misleading. Self-Knowledge Guided Retrieval Augmentation itself found direct self-assessment prompting to be the weakest strategy it tested. We include SKR-style nearest-neighbour gating as a baseline for the claim-only branch.

Proposed Experiments and Decisive Checks

We evaluate on SciFact and SciFact-Open, ClimateCheck, ClimateFEVER, and AVeriTeC, using the Conflicting label to test conflict detection. We add controlled noise injection on FEVER and Feverous, inserting distractor documents up to five times the gold count, following ATTUN's protocol. Metrics are macro-F1, the area under the cost-accuracy curve with cost counted in large-model calls and measured wall-clock time, and conflict-detection F1 on AVeriTeC.

The first decisive check is a per-claim oracle audit. Among claims the small model gets wrong, we record what fraction are fixed by repair alone, by escalation alone, by both, or by neither. This determines whether the three-way decision has any value before any router is trained.

The second check is a noise sweep. As distractors are added, a scalar router should escalate more often with flat accuracy, while $S$ should rise and the repair action should absorb the extra traffic.

The third check compares routers trained on outcome-grounded labels against routers trained on 70B teacher usefulness labels. It tests whether teacher labels reproduce the lexical-mismatch errors seen on ClimateCheck.

The fourth check examines the conflict case on AVeriTeC and SciFact-Open, confirming that claims with genuinely conflicting evidence are labeled Conflicting and not sent for repair.