Abstract
For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.
Introduction
Most Neural Machine Translation (NMT) systems translate sentences in isolation, effectively ignoring the surrounding context. This isolation leads to "discourse errors" where a translation is technically fluent but contextually incorrect. For instance, the English pronoun it might be translated into French as masculine (il) or feminine (elle) depending on the gender of the noun it refers to in a previous sentence.
This research addresses two major gaps in the field. First, an evaluation gap: standard metrics like BLEU are context-blind, rewarding word-overlap while failing to penalize a single wrong pronoun that can ruin a paragraph's meaning. Second, a modeling gap: while multi-encoder architectures have been proposed to incorporate extra-sentential context, it has remained unclear which specific architectures actually improve discourse-level performance.
Architectures for Contextual NMT
The study compares several NMT architectures designed to bridge the gap between isolated sentence translation and document-level awareness:
- Baselines: Standard sequence-to-sequence (S2S) models and simple concatenation methods like
2-TO-2, where the previous and current sentences are joined into a single input string. - Multi-Encoder Models: These architectures use a separate encoder for the previous sentence. The context is combined using methods such as concatenation, attention gating, or hierarchical attention.
- The Novel S-HIER-TO-2 Strategy: This strategy uses multiple encoders for source context but is uniquely trained to decode both the previous and current target sentences. This approach forces the decoder to maintain internal consistency with its own previous outputs.
Probing Discourse with Contrastive Test Sets
To move beyond BLEU, the authors introduced two hand-crafted benchmarks using contrastive pairs. In this setup, a model is presented with a correct translation and a carefully crafted incorrect one (e.g., a pronoun with the wrong gender). If the model assigns a higher probability to the correct version, it passes the test.
- Coreference Set: 200 pairs testing the translation of it and they based on antecedents in the previous sentence. This includes "semi-correct" cases where the model must match a potentially "wrong" (but already established) translation of a noun to maintain consistency.
- Coherence/Cohesion Set: 100 blocks testing lexical disambiguation (choosing the right synonym based on context) and alignment (repeating the same word used previously).
Results and Discussion
The experiments, conducted on the English-French OpenSubtitles2016 dataset, revealed that source context alone is insufficient. Standard multi-encoder models that only looked at the previous source sentence performed no better than a random guess (50%) on the coreference test.
In contrast, the S-HIER-TO-2 model achieved a significant accuracy of 72.5% on coreference. This suggests that to translate a pronoun correctly, the model needs to know the gender of the French noun it chose in the previous sentence—the target-side context—rather than just the English source word.
While the S-HIER-TO-2 model also improved coherence/cohesion accuracy to 57% (up from 50%), the relatively lower score highlights that lexical consistency remains a more difficult challenge than pronoun agreement. Notably, these discourse improvements did not come at the cost of general quality, as the S-HIER-TO-2 model showed a +1.0 BLEU gain over the baseline across various film genres.
Conclusion
This research demonstrates that standard multi-encoder NMT models often fail to capture discourse phenomena despite improvements in automatic metrics. The success of the S-HIER-TO-2 model highlights the critical importance of target-side context. Future work should look toward "stream decoding," where contextual information is maintained across entire documents to handle long-range dependencies beyond a single previous sentence.
Summary updated