Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation

Xinglin Lyu, Junhui Li, Zhengxian Gong, Min ZhangEMNLP 2021 · OpenReview

Xinglin Lyu, Junhui Li, Zhengxian Gong, Min Zhang. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Summary

This research introduces a novel approach to ensure that technical terms and key nouns are translated consistently throughout a document in Neural Machine Translation (NMT). By identifying "word links" across sentences and employing a specialized cross-sentence attention mechanism combined with a consistency-focused loss function, the authors achieve state-of-the-art translation quality and significantly higher lexical consistency compared to standard models.

The Problem: Lexical Inconsistency

While sentence-level NMT has become highly proficient, it often suffers from a lack of "discourse awareness." A common frustration in document-level translation is lexical inconsistency: a specific term (e.g., "real estate sector") might be translated one way in the first paragraph and a completely different way in the third.

Example of lexical inconsistency where the same source term is translated into multiple English variants by sentence-level NMT.

In human translation, the principle of "one translation per discourse" usually holds—if a term is chosen for a concept, it is used consistently. Standard NMT models, even those that look at surrounding context, often fail this. Previous attempts to fix this used "caches" to remember past translations, but these are risky; if the model makes one wrong translation early on, the cache "poisons" the rest of the document. This paper seeks a "softer," more robust way to encourage consistency during the encoding phase.

The authors propose a three-pronged strategy integrated into the Transformer architecture:

  1. Word Linking: Before translation, the system identifies "words of interest" (recurring non-stop words). It creates a "link list" for each word, mapping all the positions where that specific word appears across the entire document.
  2. Word-Link-Attention Sub-Layer: The authors insert a new sub-layer into the Transformer encoder. While standard self-attention looks at words within the same sentence, this layer allows a word to "talk" to its linked counterparts in other sentences. This allows the model to share context information globally across the document.
  3. Consistency Constraint Loss ($J_{CC}$): To move beyond just sharing information, they introduce an auxiliary loss function. This function penalizes the model if the hidden representations (vectors) of linked words are too different from each other. By forcing these vectors to be closer in the semantic space, the decoder is naturally nudged to produce the same output word for each instance.
Architecture of the proposed Transformer encoder block incorporating Word-Link Attention and cross-sentence context.

Crucially, this model translates sentences synchronously. Unlike cache-based methods that must go sentence-by-sentence, this approach can process the whole document at once.

Evaluation and Key Insights

The researchers evaluated their approach on several major translation tasks, including Chinese $\leftrightarrow$ English (NIST and IWSLT datasets) and English $\to$ French. They also introduced a new metric, the Lexical Translation Consistency Ratio (LTCR), which calculates the percentage of identical translation pairs for a given source word within a document.

  • The "Consistency Gap": In human-translated Chinese-English documents, consistency (LTCR) is about 74%, while standard NMT models hover around 43%. This confirmed a massive room for improvement.
  • Nouns Matter Most: Consistency is naturally higher for nouns than for verbs or adjectives. The model's improvements were most pronounced where they mattered most: technical terms and entities.
  • Soft Constraints > Hard Caches: By using a loss function rather than a hard-coded cache, the model remains flexible. If a word should be translated differently due to a specific local context (like a pun or a change in part-of-speech), the model isn't strictly forbidden from doing so.

Key Results

  • Translation Quality: The proposed model achieved a significant boost in BLEU scores. On Chinese-to-English News, it improved BLEU by +2.23 over the baseline Transformer.
  • Consistency Gains: The LTCR saw dramatic improvements. On the TED dataset, consistency jumped from 52.51% to 62.79%.
  • Human Preference: In a blind human evaluation of 500 sentences, judges preferred the "Word-Link" translations in 64% of cases where a preference was expressed, citing better coherence and term stability.

Limitations

The extra attention layers and the dual-encoding required for the consistency loss increase training time by about 39%. Additionally, the method depends on accurately identifying "words of interest" and their links, which usually requires stemming or morphological analysis. Errors in this preprocessing stage could propagate through the model.

Summary updated

BibTeX
@inproceedings{lyu2021encouraging,
  author = {Xinglin Lyu and Junhui Li and Zhengxian Gong and Min Zhang},
  title = {{Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=9FZtTDhT_s8},
}
Figure