Measuring Discourse Coherence in Document Level Machine Translation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2125 papers205 conceptsfrozen 2026-10-04

For decades, machine translation has operated under the "sentence-as-an-island" paradigm. Systems translate one sentence, discard the memory, and move to the next. While this works for isolated phrases, it fails for documents where meaning is woven across sentence boundaries through pronouns, specialized terminology, and logical flow.

Recent research has shifted toward Document-Level Machine Translation (DocMT), which attempts to preserve this "connective tissue" of language. However, a significant hurdle remains: our standard tools for measuring success are fundamentally blind to the very improvements we are trying to achieve.

The Contextual Blind Spot

The core challenge of DocMT is resolving inter-sentential dependencies. As Jörg Tiedemann and Yves Scherrer highlighted in Neural Machine Translation with Extended Context, traditional models "forget" the gender of a subject or the specific technical term used three sentences ago.

This leads to three primary types of discourse errors:

  1. Anaphora and Coreference: Using the wrong pronoun (e.g., "it" vs. "she") because the antecedent was in a previous sentence.
  2. Lexical Consistency: Translating the same technical term differently across a document, confusing the reader.
  3. Ellipsis: Failing to fill in words that are omitted in the source but required in the target language (common in "pro-drop" languages like Japanese or Chinese).

While models are getting better at handling these—for instance, through "concatenation" methods explored by Sharid Loáiciga in Analysing concatenation approaches to document-level NMT in two different domains—the metrics used to evaluate them often fail to notice.

Why BLEU is Blind to Coherence

The industry standard for evaluation, BLEU, measures local n-gram overlap. It checks if the machine's words match the reference's words in small windows. This is a "surface-level" check.

As Yirong Sun argues in Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation, these metrics are often inversely correlated with human preference in DocMT. A model might produce a highly coherent, stylistically consistent document that uses slightly different phrasing than the reference. BLEU will penalize this "creative" but superior translation while rewarding a disjointed, sentence-by-sentence translation that happens to match the reference's word choice.

This "Evaluation Gap" means that linguistically motivated enhancements—like the anaphora resolution models proposed by Ronan Le Nagard in Aiding Pronoun Translation with Co-Reference Resolution—often show zero improvement in BLEU scores, despite being clearly better to human readers.

Beyond Word Matching: New Evaluation Frameworks

To bridge this gap, researchers are developing metrics that look at the "logic" of a document rather than just its words.

The Path Forward: Zero-Shot and Large Context

The field is moving toward models that treat the document as a single unit. Biao Zhang demonstrated in Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents that we can even teach models to translate documents in "resource-poor" languages by leveraging document-level data from "resource-rich" ones.

The intuition is simple: if a model learns how to maintain a "story" in English and French, it can transfer that structural understanding to other languages, even if it has only seen isolated sentences in those languages before. As we move toward ultra-long context windows and more sophisticated discourse metrics, the goal is to move MT from a tool that translates words to a tool that translates meaning across entire narratives.

Papers in this version

Showing 60 of 125 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 87 papers
  2. Version 1 · 2026-10-04 · 125 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_document_level_evaluation_gap_0b6a1db9,
  title = {document-level evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-level-evaluation-gap/rdr_0b6a1db9cab84d4fb76e3f82e7d48c69},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_measuring_discourse_coherence_in_documen_0b6a1db9_v1,
  title = {Measuring Discourse Coherence in Document Level Machine Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/measuring-discourse-coherence-in-document-level-machine-translation/rdr_0b6a1db9cab84d4fb76e3f82e7d48c69/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}