Automated fact-checkers are supposed to decide whether a claim is true by reading evidence. In practice they often guess from the wording of the claim alone. A well-known English study found that a model shown only the claim, with no evidence, reached 61.7% accuracy on the FEVER benchmark by picking up cue phrases such as "did not" and "never" (Towards Debiasing Fact Verification Models). In the 25-language X-FACT benchmark the gap is starker: a claim-only model reaches 38.2% macro F1 in-domain, against 41.9% for the best evidence-using model (X-Fact). When most of the verdict can be read off the claim's surface, the evidence spans and explanations these systems produce say little about how they actually decided. Nobody has measured, language by language, how much of a multilingual fact-checking benchmark can be solved this way, or what these shortcuts look like in each language.
This proposal starts from one observation. In English, the shortcut cues are short runs of adjacent words, so counting word pairs finds them. In languages that put the verb last or build words from stacked suffixes, the same cue is often a suffix on a verb at the end of the clause, or two words far apart in the sentence. Bag-of-phrases diagnostics built for English will under-count these cues, and so will debiasing methods built on those diagnostics. We propose a cheap, automated audit that works across languages. It uses weighted word-count classifiers (TF-IDF, which scores each term by how often it appears in a claim, discounted by how common it is across all claims) over three feature types: adjacent word sequences, character sequences, and pairs of words linked by a grammatical relation regardless of their distance. The audit produces a quantitative "artifact section" for dataset documentation. We will test whether it predicts where full fact-checking models break, and whether a debiasing method guided by it beats the English-style version on languages with different word orders.
Core Claim
Our claim is that the surface form of verdict-predictive shortcut cues depends on a language's word order and morphology. As a result, both measuring and removing these cues must use features that match that form. Concretely, we predict three things. In English and other languages where the verb comes second, adjacent-word TF-IDF features recover most of the claim-only signal. In verb-final, suffix-heavy languages, character-sequence features and dependency-pair features (a word and its grammatical head, taken from a Universal Dependencies parse and counted without regard to order) recover substantially more signal than adjacent-word features do. Reweighting training examples with a lexicon built from the matching feature type will reduce shortcut reliance on counterbalanced test claims more than the English word-n-gram reweighting of Schuster et al. does.
The novelty is not a new verifier. It is a measurement instrument, validated against model failures, that tells a dataset consumer how much of each language partition an evidence-blind model can solve and through which kind of cue. The same instrument also supplies the lexicon a debiasing method needs.
Relation to Prior Work
Three strands motivate the gap. Schuster et al. identified claim-only cues with Local Mutual Information, a score for how strongly an n-gram co-occurs with a label. They reweighted training examples accordingly, raising BERT from 58.3% to 61.6% on a hand-built Symmetric Test Set of 956 counterbalanced pairs. Their work was entirely English and used adjacent word n-grams. We reuse their method but change the feature space, and we ask whether that choice matters across typologies. X-FACT supplies the multilingual data and the evidence that claim-only signal is large there. It also shows that only 45% of its evidence snippets actually suffice for verification, which gives models a reason to fall back on the claim. It never breaks the claim-only signal down by language or cue type. CONCRETE (CONCRETE) improves cross-lingual retrieval to 19.83% zero-shot macro F1 but does not ask how much of what remains is shortcut. The survey of explainable fact-checking (Explainable Automated Fact-Checking: A Survey) found that no evaluated system produces causal explanations. Our audit gives a concrete, measurable reason why evidence-grounded explanations can be decorative: the label was reachable without the evidence.
The word-order literature constrains the design rather than supplying it. Liu et al. (On the Importance of Word Order Information in Cross-lingual Sequence Labeling) show that multilingual encoders overfit to English word order, and that reducing order sensitivity helps transfer. That suggests order-invariant features may generalize better. Arviv et al. (Improving Cross-lingual Transfer through Subtree-aware Word Reordering) show that grammatical structure, not linear position, is what survives across typologies. Our dependency-pair features follow that lesson: they keep head–dependent structure and discard linear order. Their approach of reordering sentences does not transfer to TF-IDF, which ignores order within its features anyway, so we do not reorder text.
The transparency literature supplies the output format. Datasheets (Datasheets for datasets) are written by dataset creators and rely on reflective questions. MT-Adapted Datasheets (MT-Adapted Datasheets for Datasets) move to consumer-side audits of legacy corpora, but those audits are manual and qualitative. We add an automated, quantitative section that a consumer can compute without contacting the original creators. This speaks directly to the volunteer-audit bottleneck noted in that line of work.
Research Program
The audit has three parts. The first is a claim-only probe suite per language partition. It contains logistic-regression classifiers on TF-IDF features of three kinds: word 1–3-grams, character 2–5-grams, and dependency pairs of the form (head lemma, relation, dependent lemma or morphological feature) from a Universal Dependencies parser. The second part measures an artifact exposure score: the macro F1 of the best claim-only probe minus the majority baseline, divided by the gap between the full evidence-using model and the majority baseline. A value near one means the benchmark's headroom is almost entirely reachable from the claim. The third part is a cue lexicon: the top Local-Mutual-Information features per label for each feature type. Native speakers will label these for cue category, such as negation, hedging, numbers, named sources, or publisher-specific phrasing.
Two perturbation tests separate the three feature types. First, we shuffle word order within clauses. Order-sensitive probes should degrade and dependency-pair probes should not. Second, we lemmatize and strip inflection. Character probes should lose their suffix-borne signal. The pattern of losses tells us, per language, where the cues live.
The lexicon then drives two uses. The first is Schuster-style example reweighting during verifier training, done with the lexicon from each feature type, which isolates the effect of feature choice. The second is an explanation check. We measure how often a verifier's evidence-attribution or culprit-phrase output, in the style of LOREN (LOREN), overlaps the claim-side cue lexicon rather than the evidence.
Language selection will be driven by X-FACT's actual partitions. We will pick partitions with enough claims that span verb-second (German), verb-medial with little morphology, and verb-final or agglutinative languages, confirming availability and size against the released splits before committing. For four of these languages, native annotators will build small counterbalanced test sets in the style of the Symmetric Test Set: roughly 300 claim–evidence pairs each, where the claim is minimally edited to flip the verdict while the evidence is held fixed. This set is the costliest component and the one that makes validation possible.
Baselines
The naive reference points are the majority-class predictor and the claim-only neural model already reported on X-FACT (38.2% in-domain macro F1, X-Fact). The strongest reported evidence-using verifier on this benchmark is Attn-EA+Meta at 41.9% in-domain macro F1 (X-Fact). The strongest zero-shot result is CONCRETE's 19.83% (CONCRETE). We will retrain the mBERT-based Attn-EA verifier on the same splits as our verifier under test, with identical seeds and tuning budget, so that every reweighting condition is compared against the same starting model.
The strongest existing competitor for the debiasing question is Schuster et al.'s reweighting with English-style word n-gram Local Mutual Information. It is the only published method in this material that targets claim-only cues and has a counterbalanced evaluation (58.3% to 61.6% on the English Symmetric Test Set). We will apply it natively per language with the same number of lexicon entries and the same reweighting schedule as our variants. Our typology-matched reweighting counts as beating it if it improves counterbalanced accuracy in verb-final or agglutinative languages by more than the across-seed standard deviation, while losing no more than about one point of in-domain macro F1. No non-English counterbalanced number exists yet, so our annotated sets supply the first ones.
The key ablation holds everything fixed except the feature type behind the lexicon: word n-grams versus character n-grams versus dependency pairs. A second ablation replaces parsed dependency pairs with the same pairs taken from a randomly permuted parse. This checks that grammatical structure, rather than the mere presence of co-occurring word pairs, carries the signal.
Why not just…?
Why not just train a claim-only multilingual transformer per language and report its accuracy? This gives an exposure number, and we will report it as a probe. It does not say what the cue is, so it cannot produce a reweighting lexicon or a documentation entry that a dataset consumer can inspect. If the transformer probe consistently finds far more signal than the best TF-IDF probe, our interpretable probes are missing cues. In that case we will say so and treat attribution over the transformer as the lexicon source instead.
Why not just machine-translate every claim into English and run the existing English audit? This is the most serious threat, because translation normalizes morphology and word order into the form that English n-grams already capture. Translation may also erase cues, for example by flattening an evidential or hedging suffix, or it may add translation-specific phrasing. We will run this pipeline as a full competitor. If its lexicon predicts counterbalanced failures and supports reweighting as well as the native typology-matched audit, the right conclusion is that translation is the cheaper tool, and the native audit is needed only where translation loses signal. We will report the per-language comparison either way.
Why not just fix the data by retrieving better evidence? Better evidence retrieval, as in CONCRETE, is complementary and does not remove the shortcut. A model can still prefer claim cues when they are cheaper. The evidence-swap test below measures shortcut reliance directly, whatever the retrieval quality. The swap also bounds how much of the result is caused by X-FACT's insufficient snippets, as opposed to the cues themselves.
Proposed Experiments
The first experiment computes artifact exposure and cue-type profiles for every selected X-FACT partition. We predict that the character and dependency-pair probes beat the word n-gram probe by a clear margin in verb-final or agglutinative languages, and roughly tie with it in English and German. If the three probe types tie everywhere, the typology hypothesis is wrong, and the contribution reduces to a documentation tool.
The second experiment tests whether the audit predicts failures. We score each claim in the counterbalanced sets by the cue-lexicon mass in its original wording. We then test whether high-score claims are the ones the full verifier gets wrong after the minimal flip, using item-level logistic regression within each language. This avoids correlating across only four languages. In the evidence-swap test, we pair each claim with evidence from a random other claim and measure how much in-domain macro F1 survives. Survival close to the claim-only probe's score confirms that the evidence channel is largely unused.
The third experiment compares reweighting variants on the counterbalanced sets and the original in-domain split, with the baselines and ablations above.
The fourth experiment measures the overlap between the verifier's highlighted spans and the claim-side cue lexicon, before and after reweighting. The explanation claim holds only if this overlap falls while counterbalanced accuracy rises.
The deliverable is a set of per-language artifact sections, in the consumer-centric format of MT-Adapted Datasheets, released with the probes and annotated sets, so that others can rerun the audit on new benchmarks such as the multilingual claim-retrieval data used in Duluth at SemEval-2025 Task 7.
The decisive check is the second experiment. If typology-matched cue scores do not predict counterbalanced failures better than English-style word n-gram scores, or better than translate-then-audit scores, the central claim fails regardless of how the probes compare.
References
- Towards Debiasing Fact Verification Models
- X-Fact: A New Benchmark Dataset for Multilingual Fact Checking
- CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval
- Explainable Automated Fact-Checking: A Survey
- On the Importance of Word Order Information in Cross-lingual Sequence Labeling
- Improving Cross-lingual Transfer through Subtree-aware Word Reordering
- Datasheets for datasets
- MT-Adapted Datasheets for Datasets: Template and Repository
- LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
- Duluth at SemEval-2025 Task 7: TF-IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrieval