Cross-lingual summarization means reading a document in one language and writing its summary in another, for example a Chinese news story summarized in English. There are two ways to do it. A pipeline translates and summarizes in separate steps. An end-to-end model does both at once. The field's case for end-to-end models is that they avoid compounding translation errors and write in natural target-language phrasing instead of stiff "translationese." At the same time, researchers increasingly score these systems with an LLM-as-a-judge, meaning a large language model prompted to rate outputs. Two findings in the literature collide here. Multilingual LLM judges prefer machine-translated text over human-authored text (Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck). Cross-lingual summaries also contain factual errors more than half the time, including error types that only arise across languages, such as wrong tense, wrong number, or mistranslated terminology (Evaluating Factuality in Cross-lingual Summarization). If both hold at once, an LLM judge could systematically rank pipeline systems above end-to-end ones because their output reads like translation, while missing the factual errors that actually matter.
We propose to test this directly with a factorial contrast set. Each source document is paired with summaries that vary independently along two axes: surface style (natural target-language phrasing versus translationese) and factual status (faithful, or carrying one controlled error from the cross-lingual taxonomy). We then measure how much of a judge's score is driven by style rather than by faithfulness. We also test one specific remedy, source-anchored verification: the judge must align each summary claim to a span of the source-language document and check only that alignment, never the summary's overall fluency. The result is either a validated evaluation protocol for end-to-end cross-lingual summarization or a clear negative finding that current judges cannot fairly compare end-to-end systems with pipelines.
Core Claim
Holistic LLM judges applied to cross-lingual summaries confound translationese with quality. The measurable signature is an inversion: the judge prefers a translationese summary that contains a cross-lingual factual error over a natural-phrasing summary that is fully faithful to the same source. We claim three things. First, this inversion rate is substantially above zero for frontier judges. Second, it is largest for tense, plural, and terminology errors. Third, it shrinks when the judgment is forced to run through explicit source-span alignment, and the shrinkage is not explained by simply giving the judge the error taxonomy.
For a judge $J$, write $s^{nat}_{0}$ for a faithful natural-phrasing summary and $s^{tr}_{e}$ for a translationese summary carrying error type $e$. The inversion rate is
$$I_e(J) = \Pr\big[\, J(s^{tr}_{e}) > J(s^{nat}_{0}) \,\big],$$
estimated over source documents with presentation order counterbalanced. A judge that tracks faithfulness has $I_e \approx 0$. A judge that tracks only style approaches the rate at which it prefers translationese when both summaries are faithful.
Relation to Prior Work
Several papers motivate the gap. Evaluating Factuality in Cross-lingual Summarization built Fact_CLS, a human-annotated factual audit of six systems in English-to-Chinese and Chinese-to-English with a ten-category error taxonomy. It found that FactCC, DAE and mBARTScore correlate poorly with humans, but it never tested LLM judges. We reuse its taxonomy as the source of our controlled errors. Understanding Translationese in Cross-Lingual Summarization showed that machine-translated references reward systems that imitate translation artifacts, which it calls an "evaluation mirage." Its evidence came from human evaluation on WikiLingua Arabic-to-English, where humans gave human-translated documents an overall quality of 2.23 against 1.74 for machine-translated ones. That paper concerns references and overlap metrics. We ask whether the mirage reappears inside the judge itself. Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck found the judge-side preference for machine-translated text and trained DIBJudge to remove it. It reports an 80% average reduction in bias severity on BELEBELE, but it does not study a judge reading a source in one language and a summary in another, nor does it separate style from factual error. We use DIBJudge as a competing remedy, not as our method.
Other work constrains the design. How Reliable is Multilingual LLM-as-a-Judge? found average cross-language agreement around Fleiss' kappa 0.3 across 25 languages. Its best case was about 0.54 for GPT-4o on WikiLingua with yes/no questions, and agreement never exceeded 0.32 on 1–5 grades. That evidence pushes us toward binary, claim-level decisions and away from Likert scores. Its setting is symmetric (same-language input and output), which is exactly the asymmetry we add. Cross-lingual Evaluation of Multilingual Text Generation found G-Eval-style judges correlating weakly or negatively in non-English generation. It therefore routes evaluation through an English reference and embedding or COMET similarity (XESE/XETQ), reaching system-level Spearman above 0.90 with ROUGE-2 on CrossSum. That number measures agreement with an overlap metric, not with human factuality judgments, which is why we treat XE as a baseline to beat rather than as a solved problem. The length-bias literature summarized in A Survey on LLM-as-a-Judge is a further confound. So is the verbosity of zero-shot LLM summarizers documented in Zero-Shot Cross-Lingual Summarization via Large Language Models: 183.7 tokens against 36.1-token references on CrossSum. We length-match every contrast pair so that style is not secretly length.
The end-to-end versus pipeline comparison itself comes from NCLS: Neural Cross-Lingual Summarization and WikiLingua. Both argue for direct models on ROUGE. Our study asks whether the LLM judges now replacing ROUGE preserve or reverse that verdict.
Research Program
Contrast set. We build it from ConvSumX (Revisiting Cross-Lingual Summarization), whose English-to-Chinese, English-to-French and English-to-Ukrainian references were annotated directly from the source rather than through a pipeline. We also use the Fact_CLS documents. For each source document we create two faithful summaries of matched length and content. The natural one is the direct ConvSumX reference, lightly edited by bilingual annotators where needed. The translationese one is produced by machine-translating a source-language summary and verified by annotators to carry the same content. From each faithful summary, bilingual annotators then make one minimal edit per error category: tense, plural, terminology, entity, predicate, and particulars (numbers or dates). Each edit is checked by a second annotator. The outcome is roughly fourteen summaries per source that differ by one controlled factor at a time.
Real-system study. Separately, we generate outputs from an end-to-end mBART-style model, a direct GPT-class zero-shot summarizer, and translate-then-summarize and summarize-then-translate pipelines. Bilingual annotators label these outputs at the claim level using the Fact_CLS protocol. This tests whether the effects found in the contrast set actually flip system rankings.
Remedy. The judging procedure has three stages. The judge splits the target-language summary into atomic claims. It quotes, for each claim, the supporting span from the source-language document, or declares that none exists. It then decides, for each claim-span pair, whether the claim is supported, checking the taxonomy categories explicitly. The summary score is the fraction of supported claims, with no holistic fluency step. We vary the prompt language across source language, target language and English, because no prior work has examined this choice for asymmetric inputs.
Baselines
The naive reference point is ROUGE-L against the ConvSumX reference. On the contrast set it should prefer whichever style the reference shares, so it calibrates how much "style detection" any metric gets for free.
The strongest non-LLM competitor is the XE protocol of Cross-lingual Evaluation of Multilingual Text Generation: an English reference bridge plus XESE (LaBSE/SONAR similarity) and XETQ (COMET). We adapt it to cross-lingual output by comparing the target summary against an LLM-generated reference. The reported number is system-level Spearman above 0.90 with ROUGE-2 on CrossSum. On human factuality labels, no reported result was found on Lacuna. We expect XE to show low inversion on style but poor sensitivity to tense and plural edits.
The strongest LLM competitor is a holistic GPT-4o-class judge in the style of G-Eval, given the source document and a factuality rubric. For context, the survey reports GPT-4-turbo at roughly 68–69% agreement with human pairwise preferences (A Survey on LLM-as-a-Judge), and How Reliable is Multilingual LLM-as-a-Judge? reports about 0.54 kappa for GPT-4o on WikiLingua yes/no questions.
The strongest debiased judge is DIBJudge, run as released. Its reported result is an 80% average bias reduction on BELEBELE. No reported result on cross-lingual summarization was found on Lacuna.
Two pipeline baselines follow the back-translation route that Fact_CLS found best among older metrics. One runs FactCC and DAE on the summary back-translated into the source language. The other translates the summary into English and applies an English LLM judge.
All LLM judges use the same backbone model, temperature zero, the same few-shot budget (at most three examples, drawn from a held-out split), and the same counterbalanced A/B ordering.
The key ablation is the same judge given the full taxonomy as a checklist but without the span-quoting requirement. This isolates whether anchoring to the source, rather than knowledge of the error types, does the work. A second ablation keeps span quoting but drops the taxonomy. Beating the competitors means two things. On the contrast set, the anchored judge must reach a mean inversion rate below half of the holistic judge's, with tense, plural and terminology detection at least ten points above the checklist ablation. On the real-system study, its claim-level agreement with human labels must exceed both XE and DIBJudge, with confidence intervals bootstrapped over documents.
Why not just…?
Why not just back-translate the summary and use monolingual factuality metrics? Evaluating Factuality in Cross-lingual Summarization already ran this route. It beat native multilingual metrics but still correlated poorly with humans, especially Chinese-to-English. Back-translation into a tenseless language can also erase exactly the tense and plural errors we inject. Our contrast set measures this loss directly: if back-translated FactCC detects tense edits as well as the anchored judge does, our remedy adds nothing for that category, and we will report it.
Why not just use XE-style embedding and COMET similarity? These metrics are cheap and avoid the judge's own translationese preference, which makes them a real threat on the style axis. The evidence for them, however, is agreement with ROUGE-2, and sentence embeddings are built to be insensitive to small surface changes such as a tense or number flip. The per-category detection rates on the contrast set will show whether that insensitivity is real. If XE matches the anchored judge on factual detection, we will recommend XE plus a cheap tense and number check instead.
Why not just prompt the judge with the error taxonomy? This is the most plausible threat, and it is why the checklist-only ablation exists. A checklist tells the judge what to look for but still lets it score the summary as a fluent whole. Span quoting forces every verdict onto a specific source passage, which should starve the fluency pathway. If the checklist alone closes the inversion gap, the contribution narrows to the diagnosis plus a prompting recommendation, and we will say so.
Why not just use DIBJudge, which already removes translationese bias? DIBJudge was trained and evaluated on same-language reading comprehension, summarization and preference data. Removing a preference for translated text is also not the same as detecting a terminology error. If DIBJudge achieves low inversion but weak per-category detection, the two remedies are complementary, and we will test using its score as a style-invariant prior combined with anchored verification.
Decisive Checks
The first check is whether holistic judges show a nonzero style preference at all when both summaries are faithful and length-matched. If they do not, the inversion story collapses and the project reduces to a cross-lingual factuality benchmark for judges.
The second check is whether, on real systems, ranking by the holistic judge places pipelines above end-to-end models where human claim-level faithfulness ranks them the other way. Ranking by the anchored judge should restore the human order. We test this with document-bootstrapped Kendall tau against human rankings, separately for each language pair.
The third check is the prompt-language comparison, especially for Ukrainian targets. The inversion rate should be highest there if the English-centric explanation for translationese bias is correct, and this fills the asymmetric-input gap left open by the multilingual reliability study.
References
- Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
- Evaluating Factuality in Cross-lingual Summarization
- Understanding Translationese in Cross-Lingual Summarization
- How Reliable is Multilingual LLM-as-a-Judge?
- Cross-lingual Evaluation of Multilingual Text Generation
- A Survey on LLM-as-a-Judge
- Zero-Shot Cross-Lingual Summarization via Large Language Models
- NCLS: Neural Cross-Lingual Summarization
- WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization
- Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation