Modeling Consistency Preference via Lexical Chains for Document-level Neural Machine Translation

Xinglin Lyu, Junhui Li, Shimin Tao, Hao Yang, Ying Qin, Min ZhangEMNLP 2022 · OpenReview

Introduction

In document-level Neural Machine Translation (NMT), maintaining lexical consistency—ensuring that repeated source-language terms are translated consistently throughout a document—is a significant challenge. Standard NMT models often process sentences in isolation, leading to "lexical translation inconsistency," where the same source word is translated into various synonyms across different sentences. While previous document-level models have attempted to address this, they often rely on "hard constraints" that can lead to over-correction, ignoring necessary contextual or grammatical variations. This paper proposes a novel approach that models consistency as a "preference" using lexical chains and a Conditional Variational Autoencoder (CVAE).

Methodology

The proposed architecture focuses on modeling consistency softly through three primary components: a consistency-aware encoder, a latent variational module, and a consistency-aware decoder.

Lexical Chains and Consistency-aware Encoder

The process begins by identifying "lexical chains," which are sets of identical words repeated throughout the source document. To leverage these chains, the researchers modified the standard Transformer encoder by adding two specialized attention sublayers:

  1. General-Context Attention: This layer utilizes a special [cls] token to facilitate information sharing across all sentences within a document.
  2. Lexical-Consistency Attention: This layer allows words belonging to the same lexical chain to interact directly, regardless of their distance in the document, effectively bridging long-range dependencies.
The Consistency-aware Encoder architecture featuring Lexical-Consistency Attention and Lexical Chains.

Latent Variational Module (CVAE)

The core innovation is the use of a CVAE to model consistency as a probabilistic preference. For every lexical chain, the model learns a consistency-tailored latent variable ($z$). During training, the model observes both source words and their target translations to learn the distribution of a "consistent preference." At inference time, the model predicts this preference based solely on the source chain.

The Latent Variational Module (CVAE) architecture, showing the interaction between Prior/Posterior networks and lexical chains.

Consistency-aware Decoding

During the translation phase, the decoder performs a "dynamic query" over the learned latent variables. This allows the model to pull in relevant consistency information at each decoding step, guiding word choice without strictly forcing a specific output. This "soft" approach ensures that the model remains flexible enough to adapt to context-specific requirements.

Experimental Evaluation

The model was evaluated across three major document-level translation tasks:

  • Chinese → English (News): LDC corpora (~0.8M sentence pairs).
  • Chinese → English (TED Talks): IWSLT data (~0.3M sentence pairs).
  • French → English (TED Talks): IWSLT data (~0.2M sentence pairs).

Additionally, a large-scale version of the News task was tested using 2 million additional sentence pairs for pre-training.

Results and Analysis

The proposed model demonstrated significant improvements over the standard Transformer baseline and several state-of-the-art document-level models, such as HAN and G-Transformer.

  • Translation Quality: On the ZH → EN (News) task, the model achieved a BLEU score of 43.27, representing a +2.72 gain over the baseline (40.55).
  • Consistency Metrics: Using the Lexical Translation Consistency Ratio (LTCR), the researchers observed a jump from 56.87% to 64.56% on the News task.
  • Human Evaluation: In blind tests, human annotators preferred the proposed model's translations in 66% of cases where differences were identified, citing better consistency and flow.
  • Interpretability: Visualizations confirmed that latent variables captured specific semantic information. For instance, when predicting the word "ecological," the query module assigned high similarity to the latent variable associated with the Chinese word for "ecology" (生态).

Conclusion and Future Work

By treating lexical consistency as a preference rather than a hard rule, the proposed CVAE-based model significantly improves document-level NMT quality. However, the current model is limited to exact word matches for lexical chains. Future work will explore the inclusion of synonyms and morphological variations to create more robust "synonym-based lexical chains." Additionally, the authors aim to refine evaluation metrics to better capture the correctness of consistent translations, rather than just the frequency of repetition.

Summary updated

BibTeX
@inproceedings{lyu2022modeling,
  author = {Xinglin Lyu and Junhui Li and Shimin Tao and Hao Yang and Ying Qin and Min Zhang},
  title = {{Modeling Consistency Preference via Lexical Chains for Document-level Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=ueCocNagOqT},
}
Figure