Automated fact-checkers are increasingly asked to explain their verdicts, and those explanations are now usually checked by another language model acting as a judge. When the claim and its evidence are in a low-resource language or a dialect, the explanation is often written in English and the judge works on a machine translation of the evidence. That is where measurement breaks down. If the translation drops a negation or garbles a name, the judge will mark a correct explanation as unfaithful, or a wrong one as faithful, and the error is blamed on the fact-checker. We propose to measure explanation faithfulness across languages the way older statistical translation handled missing parallel data. Instead of trusting one translation path, the judge routes the evidence through several bridge ("pivot") languages, including a closely related standard language where one exists. It then combines the per-path judgments. Where the paths disagree, it flags that item as a probable translation failure rather than a reasoning failure.
In two sentences: a faithfulness judge for cross-lingual fact-checking should not rely on a single translation of the evidence, because translation errors then pass as explanation errors. Judging each explanation claim through several independent pivot paths, and treating disagreement between paths as a signal, should give scores that agree better with bilingual humans and should say which component actually failed.
Core Claim
The claim is about measurement, not about building a better fact-checker. For explanations of claims in non-English languages and dialects, a structured judge has two stages: it splits the explanation into atomic claims, then matches each one against evidence reached through several pivot paths. We expect this judge to agree more closely with bilingual human annotators than three alternatives: a direct multilingual judge, a judge given a single English translation, and a single-path judge sampled repeatedly. We also expect the gap to widen as the source language moves away from high-resource standard varieties. A second, separable claim is that disagreement between paths can be used to separate translation-induced errors from errors in the fact-checker's reasoning. This is the cross-lingual counterpart of the finding in A Closer Look at Claim Decomposition, which showed that errors made while splitting claims into atomic facts get charged to the system being evaluated.
Why This Gap Is Real
Each of the three directions stops short of this problem. Two papers show that LLM judges of fact-check explanations are both necessary and fragile, and both are English-only. Can LLMs Produce Faithful Explanations For Fact-checking? refines explanations through multi-agent debate on PolitiHop and scores them with G-Eval, a single prompted GPT-4 Turbo judge. It reports a G-Eval sentence faithfulness of 4.99/5, yet human annotators found only 30% of the same system's explanations fully faithful. A single holistic judge is therefore far too lenient even in English, and this finding motivates our work. VALOR-EVAL supplies the method we reuse: it scores faithfulness (precision) separately from coverage (recall) by first extracting structured units with an LLM and then matching them semantically. It reports 82.35% hallucination-detection accuracy against 5.88% for the dictionary metric CHAIR, but only for English image descriptions.
Cross-lingual fact-checking work has not measured explanations at all. CONCRETE trains a cross-lingual retriever and reaches 19.83% zero-shot Macro F1 on X-Fact. Claim Matching Beyond English matches messages in Indian languages to existing fact-checks. Neither produces or scores rationales.
The pivot-translation literature supplies the principle. Machine Translation by Triangulation and Pivot language approach for phrase-based SMT sum over intermediate phrases instead of committing to one serial cascade, which reduces compounding errors. Related-language pivots, such as Mandarin for Cantonese in Leveraging Mandarin as a Pivot Language, preserve structure that a distant pivot like English loses.
One finding constrains the design. Triangulation assumes the target depends on the source only through the pivot, and polysemy breaks that assumption. For this reason every path in our judge keeps the source text visible, and pivots are added to the source rather than substituted for it. We test this choice directly.
Research Program
We build a judge in two stages. In the first stage, an LLM splits each explanation into atomic claims about entities, attributes, relations and polarity, following the unit scheme of VALOR-EVAL adapted to text. We audit this split with the DecompScore-style entailment check from Wanner et al., because splitting errors are a known confound.
In the second stage, for each source-language evidence passage $e$ we build $K$ translation paths $p_1,\dots,p_K$. These use at least two different pivot languages, including a related standard language when one exists (for example Standard German for Bavarian, or Mandarin for Cantonese), plus English. For each atomic claim $a$, a matcher sees the source evidence and one pivot rendering and returns a support probability $q_k(a)$. We combine the paths as
$$q(a) = \sum_{k=1}^{K} w_k\, q_k(a), \qquad u(a) = \frac{1}{K}\sum_{k=1}^{K}\big(q_k(a)-q(a)\big)^2,$$
where $w_k$ is a path weight: uniform by default, or set from reference-free translation-quality estimates in a variant. The first expression is the triangulation sum with paths in place of phrases. The disagreement $u(a)$ is the attribution signal. A claim with high $q(a)$ and low $u(a)$ counts as supported, low $q(a)$ with low $u(a)$ counts as unfaithful, and high $u(a)$ goes to a separate "translation-uncertain" bin rather than being charged to the fact-checker. Faithfulness is the fraction of explanation claims judged supported. Coverage is the fraction of gold-evidence claims, extracted the same way, that appear in the explanation.
The data plan has two parts. First, from X-Fact we select claims in languages spanning resource levels and generate explanations from several verifiers: direct multilingual prompting, translate-then-verify in English, and a retrieval-grounded verifier. Bilingual annotators label each atomic claim as supported, unfaithful, or undecidable due to translation. Second, we build a small dialect probe for Cantonese and Bavarian, because BavGround shows that LLM competence drops on regional varieties.
To obtain labels at scale without annotation, we run a controlled perturbation study. We take English claims with gold explanations from EX-FEVER and have their evidence professionally translated into a subset of target languages. We then inject known explanation errors (entity swaps, dropped negations, invented events, following the typology of Kim et al.). Separately, we inject known translation errors into individual pivot paths. This gives ground truth both for judge accuracy and for attribution.
Baselines
The do-nothing reference is ROUGE-L and BERTScore between the explanation and the gold explanation or evidence. EX-FEVER explicitly calls such overlap measures inadequate, and we include them to quantify how badly they rank systems across languages.
The strongest existing method is a holistic G-Eval judge as used in Can LLMs Produce Faithful Explanations For Fact-checking?. We run it with the same backbone model as our judge, reading the source-language evidence directly. The only reported figure is its 4.99/5 score against 30% human-judged full faithfulness on English PolitiHop. We found no reported cross-lingual agreement result on Lacuna for G-Eval or any LLM faithfulness judge.
We give the same backbone in two further configurations. The first is a structured VALOR-style judge with a single English translation produced by the strongest available translation system, which is the standard cascade. The second is a structured direct multilingual judge with no translation. All baselines share the backbone model, the prompts for the extraction stage, and a matched budget of LLM calls. The direct judge is sampled $K$ times so that it uses as many calls as our $K$-path judge. Prompt tuning for every configuration uses the same held-out development split.
Beating the baselines means a statistically significant gain in agreement with human atomic-claim labels, measured by Cohen's kappa and Spearman correlation at the system level. The gain must hold on low-resource languages and the dialect probe. It must come without a loss on high-resource languages. And the translation-uncertain bin must be meaningfully precise on the injected translation errors.
Three ablations isolate the key ingredient. In the first, all $K$ paths go through English using different translation systems, which tests whether pivot-language diversity matters beyond translation diversity. In the second, we remove the related-language pivot. In the third, pivots replace the source text instead of accompanying it, which tests the conditional-independence concern.
Why not just…?
Why not just have a strong multilingual LLM judge the source-language evidence directly? This is the real threat. On high-resource languages we expect it to match our judge, and if it does so everywhere the pivot machinery is unnecessary and we will report that. The existing evidence against it is indirect: BavGround reports degraded competence on dialects, and X-Fact reports zero-shot verification collapsing to about 15–16% F1. Our language-stratified agreement analysis, with the direct judge given an equal call budget, is designed to settle the question.
Why not just translate everything into English once and use an English judge? This is the cascade. The triangulation literature and Analyzing the Use of Character-Level Translation show that serial pipelines cannot recover from errors made at the first step. Our injected translation errors measure directly how often a one-path judge charges those errors to the explanation.
Why not just sample one judge several times and vote, as in Just Ask One More Time!? Self-agreement reduces sampling noise but not systematic misreadings of the source. If votes from a single path track human labels as well as votes across pivot paths, our diversity argument fails. The equal-budget comparison between the sampled direct judge and the English-only multi-system judge tests this.
Why not just filter bad translations with a quality estimator? Filtering discards exactly the hard, low-resource items the measurement needs. We include quality-estimate path weights as a variant, so any benefit of filtering is captured inside the method and compared against uniform weights.
Decisive Checks
The central experiment compares agreement with bilingual human labels across language tiers and the dialect probe. The key prediction is an interaction: our judge's advantage over the direct and cascade judges should grow as resources fall and should be near zero for high-resource languages. Without that interaction the claim fails.
The second check is attribution. On the perturbation set, the translation-uncertain bin should capture most injected translation errors and few injected explanation errors. We report precision and recall for both error types.
The third check asks whether measurement changes conclusions. We rank the candidate verifiers on X-Fact under each judge and test whether the cascade judge systematically penalizes systems that reason correctly from the original-language evidence. That would mean current single-path evaluation rewards the wrong systems.
Finally, we report the extraction-stage DecompScore for each language, so that any gain cannot be traced to cleaner splitting rather than to triangulation.
References
- A Closer Look at Claim Decomposition
- Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval
- X-Fact: A New Benchmark Dataset for Multilingual Fact Checking
- Claim Matching Beyond English to Scale Global Fact-Checking
- Machine Translation by Triangulation: Making Effective Use of Multi-Parallel Corpora
- Pivot language approach for phrase-based statistical machine translation
- Leveraging Mandarin as a Pivot Language for Low-Resource Machine Translation between Cantonese and English
- BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian
- EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification
- Analyzing the Use of Character-Level Translation with Sparse and Noisy Datasets
- Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios