Does the Verdict Depend on the Evidence? Counterfactual Evidence Probes for Multilingual Fact-Checking

When a language model checks a claim written in Polish, Indonesian or Arabic, two different things can make its verdict unstable. The first is that the model does not fully understand the language, so rewording the instruction flips the answer. The second is that the evidence does not actually settle the claim, so adding or removing a passage flips it. Current tools measure only the first. They reword the prompt several times and treat disagreement as uncertainty. In low-resource languages that signal is contaminated, because the model is shaky on every claim, whether or not the evidence is sufficient. We propose to separate the two sources. The model writes a step-by-step rationale that names the passages it relies on. We then test the verdict against that rationale: it should change when the named passages are deleted, and it should stay put when irrelevant passages are added. A verdict that survives the removal of its own cited support is not grounded in evidence, and the claim should be treated as unverifiable.

This matters because the hardest and most useful label in real fact-checking is "not enough evidence." Models are overconfident on it, and they are most overconfident in exactly the languages with the least training data. Benchmarks such as X-Fact show supervised systems falling from about 42% to 15–16% macro F1 on unseen languages and sources (X-Fact). A cheap, training-free test of whether a verdict actually rests on the retrieved evidence would give fact-checkers an abstention signal that does not require labeled data in every language.

Core Claim

The claim has two parts. First, verdict instability in multilingual fact-checking splits into two measurable parts. Instability under instruction rewording should rise as a language's resource level falls, and it should carry little information about whether the evidence is sufficient. Instability under evidence edits should track evidence sufficiency and be comparatively flat across languages. Second, a score built from targeted evidence edits, meaning deletion of the passages the model's own chain of thought cites plus injection of distractors, should predict "unverifiable" claims and wrong verdicts better than rewording-based sensitivity or repeated sampling at the same number of model calls. The gap should be largest on zero-shot languages.

The closest prior work plays three roles here. Chen et al.'s SENSEL (On the Relation between Sensitivity and Accuracy in In-Context Learning) supplies the method template: it perturbs instructions or example order and abstains when predictions disagree. It reports a negative correlation between sensitivity and correctness of up to −0.40 and a +4.1-point gain in area under the F1–coverage curve on English classification with GPT-J. We keep the selective-prediction setup but change what gets perturbed. We perturb the evidence rather than the wording, and we choose the edit using the model's rationale. ProSA (ProSA) motivates the decomposition. It ties prompt sensitivity to flat output distributions, meaning low decoding confidence, and low-resource languages should produce such distributions for reasons unrelated to the evidence. That is the confound we want to remove. EuroVerdict (EuroVerdict) motivates the chain-of-thought angle and also constrains it. Step-by-step reasoning gave Llama-3.1-8B-Instruct over 95% label accuracy but the worst explanation text, because the reasoning drifted and sometimes switched language. We therefore use the rationale only as a pointer to which passages to test, not as the output a fact-checker reads. Drift in the rationale then becomes something we measure, rather than something we must fix.

Research Program

Each claim comes with a native-language evidence pool. Where page text is recoverable we use full article text rather than search snippets, since X-Fact found that only 45% of snippets suffice while full pages could verify up to 83% of claims. This choice deliberately creates the noisy long-context setting studied in the contextual-distraction literature. The model is prompted to reason step by step, to cite passage identifiers for every step that uses evidence, and to output a verdict that includes an explicit unverifiable option.

From one base run we derive three perturbation families of equal size $k$.

The rewording family issues $k$ paraphrased instructions in the ProSA style. It gives an instruction-flip rate $s_P$.

The distractor family injects $k$ sets of passages retrieved for other claims in the same language and topic. It gives a distractor-flip rate $s_D$, which should be low for a well-grounded verdict.

The support-removal family deletes the passages cited in the rationale, either all at once or one at a time up to $k$ edits. It gives a removal-flip rate $s_R$, which should be high for a well-grounded verdict.

We define evidence dependence as

$$g = s_R - s_D - \beta\, s_P,$$

where $\beta$ is fit on a small development split so that $g$ is uncorrelated with $s_P$ on grounded, correctly labeled claims. The $\beta s_P$ term removes the part of $s_R$ and $s_D$ that simply reflects language-level shakiness. Low $g$ triggers abstention or an "unverifiable" verdict.

We also measure how well the rationale points at the real evidence. On CHEF, human-annotated evidence sentences let us compute the precision and recall of the cited passages against gold (CHEF). We then check whether removal flips are stronger when citations hit gold evidence. We run the whole program under two reasoning regimes. One reasons purely in the native language. The other is an English-anchored, language-mixed chain of thought following Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought, which reported reduced topic drift in Korean but never tested evidence-grounded tasks. The question is whether English anchoring lowers $s_P$ in low-resource languages without reducing $g$'s ability to separate sufficient from insufficient evidence.

Models are open-weight so that log-probabilities are available. They are Llama-3.1-8B-Instruct, for continuity with EuroVerdict, and one larger multilingual instruction model. All scores are computed without parameter updates.

Baselines

The naive reference point is a single chain-of-thought run with the unverifiable option available and no perturbation. Its selective curve comes from ranking claims by the verdict token's probability, which is the MaxProb baseline SENSEL compared against. Any method must beat it at equal accuracy on the claims it answers.

The strongest competitor for abstention is SENSEL itself (On the Relation between Sensitivity and Accuracy in In-Context Learning). We run it on the same claims, evidence and model, with the same $k$ paraphrases as our rewording family, so its model calls are a subset of ours. We also run a call-matched version that uses $3k$ paraphrases. On fact-checking with retrieved evidence there is no reported result found on Lacuna for SENSEL or any sensitivity-based abstention method. Its only reported figure is +4.1 AUC over probability baselines on English classification with GPT-J-6B.

The second strong competitor is self-consistency: sampling $3k$ chains at nonzero temperature and abstaining on disagreement. This matches our budget exactly.

The third is a distraction-aware reader in the style of DAS (Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering). It uses the "if the evidence does not settle this, return unverifiable" instruction plus reweighting of each passage by query likelihood. It reported a 64% average exact-match gain on English open-domain QA and has never been tested multilingually.

For veracity accuracy we cite reported numbers as reference points rather than rerun them. CONCRETE reports 19.83% macro F1 on X-Fact's zero-shot split, 2.23 points above the previous best (CONCRETE). Attn-EA with metadata reports 41.9% in-domain (X-Fact). On CHEF the latent joint system reports 65.26% macro F1 against a 78.47% gold-evidence ceiling (CHEF). A LoRA model fine-tuned on X-Fact's training split serves as the supervised upper reference, given the same development data used to fit $\beta$.

Three ablations isolate the key ingredient. The first replaces cited-passage removal with removal of the same number of random passages, which tests whether the rationale's citations carry information. The second drops the $\beta s_P$ correction, which tests the decomposition. The third uses $s_R$ alone or $s_D$ alone. Beating the competitors means a higher area under the F1–coverage curve and higher F1 on the unverifiable class than both SENSEL and self-consistency at matched calls, on X-Fact's zero-shot split, with a bootstrap interval over claims that excludes zero. The random-removal ablation must also lose clearly to cited removal.

Why not just…?

Why not just use self-consistency? It is the most serious threat. If sampled chains already disagree precisely when evidence is insufficient, targeted edits add nothing. Our view is that sampling variance, like rewording variance, mixes language uncertainty with evidence uncertainty. It also cannot detect the failure we care most about: a model that confidently and consistently answers from memory while ignoring the evidence. Such a model is stable under sampling but should fail the removal test. The call-matched comparison settles this directly. If self-consistency ties our score on zero-shot languages, we will report that the decomposition is not needed.

Why not just tell the model to say "unverifiable" and prune distractors? Instruction-based abstention and pruning, whether through DAS or the parallel per-passage scoring of Superposition Prompting, improve accuracy under noise. However, they depend on the very instruction-following that is weakest in low-resource languages, and the file notes that negation instructions fail on ambiguous inputs. Both were shown only in English QA. We include the DAS-style reader as a baseline and also run our probe on top of its pruned evidence. If pruning alone matches our unverifiable-class F1, the probe is redundant.

Why not just fine-tune on each language's fact-checks? EuroVerdict found LoRA beat zero-shot and chain-of-thought prompting when in-language data existed. X-Fact, however, shows supervised models collapsing to 15–16% F1 on unseen sources and languages. It also shows that adding 12,000 English PolitiFact claims made results worse, so borrowing English data is not a fix. Fine-tuning is our upper reference on in-domain languages. The question our experiments answer is whether it still wins on the zero-shot split, where no training data for the language exists.

Why not just translate everything to English? EuroVerdict found native-language evidence beat translated evidence. Translation would also hide exactly the language-dependent sensitivity we want to measure. We report a translate-then-verify condition only as a check that $s_P$ falls after translation, which is what the decomposition predicts.

Proposed Experiments and Decisive Checks

The first experiment is a controlled test on CHEF, which provides gold evidence sentences and a not-enough-information label. We build matched pairs in which each claim appears once with its gold evidence and once with the gold sentences deleted, leaving distractors in place. By construction, the second copy should become unverifiable. The model has not changed between the two copies, only the evidence sufficiency. So $g$ should drop sharply within a pair while $s_P$ barely moves. If $s_P$ moves as much as $g$, the decomposition claim is false.

The second experiment runs on X-Fact across its in-domain, out-of-domain and zero-shot splits. We report seven-class macro F1, unverifiable-class F1, and area under the F1–coverage curve per language. Languages are grouped by resource level, using the number of training claims per language in X-Fact as the proxy. The decisive pattern is a negative slope of $s_P$ against resource level, a flat slope for $g$'s predictive power, and the largest advantage of $g$ over SENSEL and self-consistency on the zero-shot split.

The third experiment runs on EuroVerdict's eight languages. It asks whether the reasoning drift reported there predicts grounding failures. Chains that switch language or wander should cite gold-irrelevant passages and show low $s_R$. If that holds, drift is the model reasoning without its evidence. We also compare native and language-mixed reasoning on $s_P$ and $g$.

The single most decisive check is the random-removal ablation. If deleting random passages works as well as deleting cited ones, the chain of thought contributes nothing, and the result is just a reinvention of evidence-dropout sensitivity. We would then report it as such.