Evaluating Discourse Phenomena in Neural Machine Translation

Rachel Bawden, Rico Sennrich, Alexandra Birch, Barry Haddow

Abstract

For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.

Introduction

Most Neural Machine Translation (NMT) systems translate sentences in isolation, ignoring the broader linguistic context. This leads to significant failures in handling discourse phenomena, such as pronoun gender agreement across sentence boundaries. For instance, translating the English pronoun "it" into French requires knowing the gender of its antecedent in the previous sentence—information that is unavailable to standard sentence-level models.

This research addresses two primary gaps: the evaluation gap, where standard metrics like BLEU fail to penalize discourse errors, and the modeling gap, where existing multi-encoder models often ignore crucial target-side context. We introduce new hand-crafted contrastive test sets and evaluate various architectures, including a novel multi-source approach that decodes both previous and current sentences.

Methods

We compared several NMT architectures to evaluate their ability to leverage context:

  • Single-Encoder Baselines: Standard NMT and "concatenation" models (e.g., 2-TO-2), where the previous and current sentences are treated as a single long sequence.
  • Multi-Encoder Models: These utilize a dedicated second encoder for the previous source sentence, combined via mechanisms like concatenation, attention gates, or hierarchical attention.
  • S-HIER-TO-2 (Novel Strategy): Our key contribution uses multiple encoders for source context but is uniquely trained to decode both the previous and current target sentences. This forces the model to maintain a "warm" internal state and direct access to its previous target-side linguistic choices.
Comparison of NMT architectures: (a) Baseline S2S, (b) Concatenate input, and (c) Multi-source S2S.

Experimental Setup

Models were trained on approximately 29 million parallel sentences from OpenSubtitles2016 (English-French), a dataset rich in conversational context. Evaluation was performed using two novel contrastive benchmarks:

  1. Coreference Test Set: 200 pairs focusing on the disambiguation of "it" and "they."
  2. Coherence/Cohesion Test Set: 100 blocks focusing on lexical consistency and disambiguation.

In these contrastive sets, the model must rank a correct translation higher than a purposefully incorrect one; a random guesser achieves 50% accuracy.

Results and Discussion

Our results demonstrate that source-side context alone is insufficient for resolving discourse ambiguities. Multi-encoder models that only looked at the previous source sentence performed no better than the random baseline (50%) on coreference tasks. In contrast, the S-HIER-TO-2 model achieved a significant improvement, reaching 72.5% accuracy on coreference and 57% on coherence/cohesion.

Performance of different models on the coreference test set over training updates.

Interestingly, simple concatenation strategies (2-TO-2) performed remarkably well, often outperforming complex multi-encoder architectures. This suggests that standard attention mechanisms are effective if the context is presented appropriately. Furthermore, we observed that while discourse-specific accuracy improved drastically, BLEU scores showed only modest gains (< 1 point), confirming that BLEU is an insensitive metric for discourse-level progress.

Conclusion

This study highlights the critical importance of target-side context in resolving discourse phenomena in NMT. By forcing the decoder to re-generate the previous sentence, models can better maintain the linguistic memory required for correct pronoun agreement and lexical cohesion. Future work will explore extending the context window beyond a single sentence and improving the computational efficiency of multi-sentence decoding strategies.

Summary updated

Abstract

For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.
BibTeX
@inproceedings{bawden2018evaluating,
  author = {Rachel Bawden and Rico Sennrich and Alexandra Birch and Barry Haddow},
  title = {{Evaluating Discourse Phenomena in Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=r1ZBpz-dbS},
}
Figure