For decades, the foundational assumption of machine translation (MT) has been that a document can be broken down into a collection of isolated sentences, each translated independently. This sentence independence assumption drastically simplifies data alignment, model training, and decoding.
However, human language does not operate in a vacuum. Meaning, style, and grammatical choices flow across sentence boundaries. When an MT model is restricted to a single sentence, it becomes "context-blind." This structural limitation introduces systematic errors that cannot be solved by simply adding more training data.
To build truly fluent translation systems, researchers have shifted toward document-level machine translation, developing mechanisms that allow models to look beyond the immediate sentence and ingest global discourse context.
The Cost of Context Blindness
To understand why sentence-level translation fails, imagine translating the following two-sentence English dialogue into French or German:
"Where is the key? I cannot find it."
If a model translates the second sentence in isolation, it has no way of knowing what "it" refers to. In French, if "it" refers to a masculine noun (le compromis), it should be le. If it refers to a feminine noun (la clé), it must be la.
Without surrounding text, a sentence-level model is forced to guess, frequently leading to critical errors in:
- Coreference and Pronoun Resolution: Dropping or misgendering pronouns because the antecedent resides in a previous sentence (Jung et al.).
- Lexical Consistency: Translating a ambiguous word like "shot" as a photograph in one paragraph and a sports goal in the next, destroying the thematic cohesion of the document (Zhao et al.).
- Discourse Relations: Missing the implicit logical connections (such as causality or contrast) that bind sentences together, resulting in a choppy, unnatural translation (Yung).
In fact, Toral et al. demonstrated that widely publicized claims of machine translation achieving "human parity" often vanish when human evaluators judge translations at the document level rather than inspecting isolated sentences.
The Challenge of Context Neglect
The most intuitive way to provide context to a modern Transformer model is concatenation: simply pasting the preceding sentences onto the current source sentence, separated by a special token (Tiedemann et al.).
While elegant, simple concatenation frequently fails due to a phenomenon known as context neglect. As Yang et al. point out, the primary sentence provides a highly concentrated, low-noise signal for translation. The surrounding context sentences, by contrast, contain a lot of information that is irrelevant to the immediate word being generated. Because standard attention mechanisms take the path of least resistance, the model learns to prioritize the local sentence and completely ignore the broader context.
To overcome context neglect, researchers have designed architectures and training objectives that force the model to rely on document-level information.
Sentence-Level MT:
[Source Sentence] ───► [Encoder] ───► [Decoder] ───► [Target Sentence]
Document-Level MT (Multi-Encoder):
[Context History] ───► [Context Encoder] ──┐
▼
[Source Sentence] ───► [Source Encoder] ──┴─► [Decoder] ───► [Target Sentence]
1. Multi-Encoder and Hierarchical Architectures
Instead of shoving all text into a single sequence, multi-encoder frameworks separate the local source sentence from the historical context. For example, Zhang et al. introduced a dedicated context encoder alongside the standard source encoder, combining their representations before decoding.
To capture long-range themes without overwhelming the model, Tan et al. proposed a hierarchical modeling approach. Their system extracts global context from the entire document and dynamically feeds it back to individual words, ensuring that local lexical choices remain consistent with the overarching topic.
2. Forcing Context Dependency via Pre-Training
If a model can translate a sentence perfectly well without looking at the context, it will continue to ignore it. To break this habit, researchers design pre-training tasks where the local sentence is intentionally broken or incomplete, making context strictly necessary for survival.
- Pre-training with Global Context (PGC): Chen et al. trained models on massive monolingual datasets by masking entire "gap sentences" or corrupting local syntax, forcing the network to reconstruct the missing pieces using only the surrounding document context.
- Divide and Rule (d&r): Dinarelli et al. artificially split sentences into arbitrary segments during training. To resolve the broken syntactic links, the model has no choice but to learn to leverage the neighboring text blocks.
3. Contrastive Learning for Coreference
Another powerful tool is contrastive learning, which explicitly penalizes context neglect. Jung et al. introduced CorefCL, a framework that generates "trick" negative examples during training. If a source text contains a gendered pronoun antecedent in a previous sentence, the system creates a counterfactual negative translation with the wrong pronoun gender. By forcing the model to maximize the margin between the correct translation and the trick example, the network is directly incentivized to track coreference chains across sentence boundaries.
Rethinking Evaluation
As translation models have evolved to process documents, traditional evaluation metrics like BLEU have become a bottleneck. Because BLEU calculates n-gram overlap on a sentence-by-sentence basis, it is blind to whether a pronoun matches its antecedent three sentences prior (Gong et al.).
To address this, metrics like Cont-COMET (Hu et al.) incorporate preceding and subsequent sentences into the evaluation pipeline. By evaluating text cohesion and document-wide consistency, these metrics align far more closely with human judgments of translation quality.
By dismantling the sentence independence assumption, document-level machine translation shifts the paradigm from a mechanical word-swapping exercise to a holistic act of interpretation—ensuring that translated text flows with the same narrative and logical continuity as the original.