Imagine reading a novel where every single sentence is grammatically flawless, yet the main character’s gender changes randomly from paragraph to paragraph, technical terms swap synonyms arbitrarily, and the logical connection between events disappears. If you evaluated this book sentence by sentence, you might give it a perfect score. If you tried to read it as a whole, it would be completely incoherent.
This paradox defines the document-level evaluation gap in machine translation (MT). While modern translation engines have grown highly adept at mapping isolated source sentences to target sentences, our automated evaluation frameworks remain largely trapped in a sentence-by-sentence paradigm. As a result, the metrics we rely on to measure progress are fundamentally blind to the connective tissue—the discourse structure—that makes a document readable.
The Illusion of Sentence-Level Success
For decades, machine translation has relied on local, segment-based metrics like BLEU, METEOR, and chrF++. These metrics operate by calculating n-gram overlap or token-level similarity between a machine-generated translation and a human reference text. Even newer neural evaluation metrics often simplify document-level quality by scoring individual segments in isolation and averaging the results across the corpus, as noted by Nitika Mathur et al. in the Results of the WMT20 Metrics Shared Task.
This averaging approach creates a severe disconnect. As Biao Zhang et al. demonstrate in Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents, local word overlaps completely fail to reflect improvements in document-level coherence or long-range linguistic dependencies. A system can achieve high local accuracy while introducing critical logical inconsistencies across sentence boundaries (Yunzhen Zhang et al.). Conversely, a model that makes sophisticated linguistic choices to preserve document-wide flow may be penalized if its choices diverge from the specific phrasing of a localized reference text (Yvette Graham).
The Core Linguistic Blind Spots
Why exactly do local metrics fail when applied to full documents? The breakdown occurs across three primary dimensions of discourse:
1. Coreference and Deixis
In many languages, pronouns and grammatical markers depend entirely on context established multiple sentences prior. For example, translating the English pronoun "it" into French requires knowing the grammatical gender of the noun it replaces, which might only appear paragraphs away.
Elena Voita et al. show that standard metrics like BLEU are fundamentally insensitive to errors in deixis, ellipsis, and pronominal anaphora. Because pronouns occur with relatively low frequency relative to the entire corpus, a model can completely scramble the gender of a main character throughout a story without causing a noticeable drop in its aggregate BLEU score (Rachel Bawden et al.; Xiao Pu 0001).
2. Lexical and Terminological Consistency
A coherent document must maintain consistent terminology. If a technical manual translates "engine" as moteur in the first paragraph, switching to propulseur in the second paragraph creates immediate confusion.
Eva Martínez García et al. point out that traditional metrics are largely insensitive to this global lexical consistency. Because they evaluate each sentence against a single reference sentence, they do not reward a system for "remembering" its past lexical choices. In fact, if a reference translation happens to use a synonym, a context-aware model that correctly enforces consistency across the document might actually be penalized by the metric for choosing the contextually superior word.
3. Discourse Relations and Information Packaging
Texts are not just collections of facts; they are structured arguments. The logical flow between sentences is often maintained through explicit or implicit discourse connectives (such as "however," "because," or "therefore").
Frances Yung argues that standard word-level metrics fail to account for the preservation of these logical relationships. When translating between languages with different discourse norms—such as Chinese, which relies heavily on implicit relations, and English, which favors explicit markers—a model must restructure how information is packaged. Kevin Duh and Hongzheng Li et al. demonstrate that when models successfully insert these implicit connectives to improve readability, traditional metrics fail to capture the resulting boost in fluency because they prioritize exact lexical matches over structural logic.
Sentence-Level Evaluation (BLEU / COMET)
[Sentence 1] -> Score: 95%
[Sentence 2] -> Score: 92%
[Sentence 3] -> Score: 94%
Total Average: 93.6% (Excellent)
*Blind to the fact that pronouns conflict and terminology shifts between sentences*
Document-Level Evaluation (Discourse-Aware)
[Sentence 1] + [Sentence 2] + [Sentence 3]
Evaluates: Coreference Chains + Lexical Consistency + Logical Flow
Total Coherence Score: Low (Broken Narrative)
Why Overlap Metrics Actively Penalize Coherence
The problem is not just that sentence-level metrics are blind to discourse; they can actively work against it. When human translators work on a document, they frequently perform non-linear transformations: they split long sentences, merge short ones, reorder paragraphs, or condense redundant information to match the stylistic expectations of the target language (Junyi Jessy Li).
Traditional metrics assume a linear, sentence-to-sentence mapping. As the authors of Adapting Sentence-level Automatic Metrics for Document-level Simplification Evaluation explain, when a model attempts these complex structural transformations, standard metrics break down because they cannot align the source and target units properly.
Furthermore, as Yirong Sun et al. observe, instruction-tuned Large Language Models (LLMs) often generate highly cohesive, fluent document translations that deviate from literal, sentence-by-sentence reference texts. Because word-matching metrics demand literal alignment, they penalize these superior, cohesive translations. This creates a bizarre dynamic where training a model to optimize for global narrative coherence can cause its traditional evaluation scores to drop (Li An).
Closing the Gap: Contrastive and Holistic Frameworks
To overcome these limitations, researchers are shifting away from open-ended n-gram matching toward evaluation frameworks designed specifically for document-level phenomena.
Contrastive Evaluation Suites
Instead of asking a metric to score an open-ended translation, contrastive evaluation tests a model's contextual awareness by presenting it with a forced choice. A targeted test suite contains a correct document translation and a corrupted variant where a single discourse feature—such as a pronoun gender or a technical term—has been altered to be contextually incorrect.
As António V. Lopes et al. and Maud Bénard et al. demonstrate, if the translation model assigns a higher probability to the correct contextual variant, it proves it is successfully utilizing long-range context. This approach bypasses the reference-matching bottleneck entirely, allowing researchers to isolate and measure specific failures in cross-sentence consistency.
Document-Wide Cohesion Metrics
Another direction focuses on designing metrics that explicitly evaluate text structure. For example, Zhengxian Gong et al. introduced metrics for Gist Consistency and Text Cohesion, which model the topic distribution and semantic flow across a document to ensure the overarching message is preserved. More recently, researchers have begun leveraging advanced LLMs as multi-dimensional judges to evaluate document-level fluency, factual tracking, and narrative voice directly, providing a more reliable reflection of human judgment than word-matching ever could (Yirong Sun et al.).
As machine translation moves deeper into long-form text generation—from scientific literature (Maud Bénard et al.) to full-length novels (Longyue Wang et al.)—closing the evaluation gap is no longer optional. Progress in document-level translation requires metrics that can see the forest, not just the trees.