Abstract
Standard Neural Machine Translation (NMT) systems often struggle with document-level consistency due to their sentence-by-sentence processing nature. This paper introduces a flexible log-linear reformulation of the noisy channel model that integrates document-level context through a dynamic merging module. By combining a standard translation model with both document-level and sentence-level language models, the proposed framework effectively handles linguistic phenomena like pronoun consistency and lexical cohesion. Evaluation on the English-Russian OpenSubtitles dataset demonstrates significant improvements on targeted linguistic challenge sets compared to context-agnostic baselines.
Introduction
While NMT has achieved remarkable success in translating individual sentences, it frequently fails to maintain coherence across a document. Issues such as inconsistent pronoun usage (deixis) or varying terminology for the same concept (lexical cohesion) arise because models lack access to the broader "big picture." A primary obstacle in addressing this is the scarcity of parallel document-level training data compared to the abundance of sentence pairs.
This research explores whether existing, high-quality translation and language models can be combined intelligently to incorporate document-level context without the need to train complex new architectures from scratch.
Methodology: Log-Linear Reformulation
The authors build upon the Noisy Channel Model, which defines translation as a balance between a target-to-source model and a language model. Using Bayes’ rule, they derive a log-linear reformulation that combines three distinct components:
- Translation Model (TM): A standard model mapping source sentences to target sentences.
- Document-Level Language Model (Doc-LM): Predicts tokens based on the entire preceding document context.
- Sentence-Level Language Model (Sent-LM): Predicts tokens based only on the current sentence.
Dynamic Merging Module
A key innovation is the Dynamic Merging Module. Rather than using fixed weights, a small auxiliary neural network determines token-by-token weights for the document context ($\alpha$) and the sentence-level bias ($\beta$). This allows the system to selectively leverage context only when linguistically necessary, such as when resolving ambiguous pronouns.
Experimental Setup
The approach was evaluated on the English-Russian OpenSubtitles dataset, where dialogue context is critical for accurate translation.
- Parallel Data: 6 million sentence pairs, including a 1.5 million subset with three sentences of preceding context.
- Monolingual Data: 30 million sequences of four consecutive sentences for training the language models.
- Challenge Sets: Targeted evaluations for Deixis (pronoun choice), Lexical Cohesion (consistent word choice), and Ellipsis.
Key Results
While overall BLEU score improvements were modest (approximately 0.21 points over the baseline), the model showed substantial gains in targeted linguistic areas. BLEU is often criticized for its inability to capture document-level nuances, making challenge sets a more informative metric for this task.
On the Deixis challenge set, the dynamic model achieved a score of 74.2%, a significant improvement over the context-agnostic baseline of 50.0%. This indicates a much higher proficiency in selecting the correct formal or informal pronouns based on the surrounding dialogue.
Discussion and Insights
The "Subtraction" Trick
The reformulation includes subtracting the sentence-level LM probability. This serves as a counter-balance, isolating the specific contribution of the document-level context and preventing it from being overshadowed by general sentence-level fluency.
Token-Specific Context Weighting
The dynamic merging module learned that context is not equally useful for all words. It assigns higher weights to the document-level LM for tokens that are highly context-dependent.
As shown in Figure 1, words like "вас" (you, formal/plural) and "тебя" (you, informal) receive higher $\alpha$ values, reflecting the model's reliance on document context to determine the appropriate social register.
Conclusion
This log-linear framework provides a modular and efficient way to incorporate document-level context into NMT. By dynamically weighting context and utilizing a "subtraction" trick to isolate document-level signals, the model achieves superior performance on linguistic consistency tasks while remaining compatible with standard beam search decoding. Future work may involve integrating multi-modal context and deeper model fusion to further enhance general translation quality.
Summary updated