Learning Efficient Lexically-Constrained Neural Machine Translation with External Memory

Ya Li, Xinyu Liu, Dan Liu, Xueqiang Zhang, Junhua Liu

Abstract

Recent years have witnessed dramatic progress in neural machine translation (NMT); however, methods for manually guiding the translation procedure remain to be better explored. Previous works proposed handling such problems through lexically-constrained beam search in the decoding phase. Unfortunately, these methods suffer from high computational complexity and "hard" beam search constraints that can generate unexpected or disfluent translations. We propose to learn lexically-constrained translation using an external memory, overcoming these disadvantages. Our model, LCNMT, encodes automatically extracted phrase pairs into an external memory, which provides constrained information during training via a memory-attention mechanism. Experiments on WMT Chinese-to-English and English-to-German tasks demonstrate the effectiveness of our approach.

Introduction

Standard NMT models often struggle with specific terminology requirements, such as technical jargon or brand names. While lexically-constrained decoding (e.g., Grid Beam Search or Dynamic Beam Allocation) forces these terms into the output, it significantly increases inference latency and often compromises sentence fluency. This paper introduces Lexically-Constrained Neural Machine Translation (LCNMT), which shifts the constraint logic from the decoding stage to the model architecture itself, allowing for efficient, high-quality translations.

The LCNMT Architecture

The core of LCNMT is a memory-augmented Transformer encoder. Instead of forcing constraints during decoding, we provide them as "suggestions" to the encoder, allowing the model to integrate them naturally into the context representation.

The LCNMT Architecture showing the integration of external memory and memory-attention layers within the Transformer encoder.

As shown in Figure 1, the architecture includes:

  1. External Memory: Stores constraint pairs (source phrase $\rightarrow$ target phrase) as Key-Value pairs.
  2. Memory-Attention Layer: A specialized layer in the encoder that allows the model to "look up" relevant target-side terminology based on the source context.
  3. Attention Loss: A supervised loss function that trains the model to attend to the correct memory slots, ensuring the constraints are effectively utilized.

Automatic Constraint Extraction

To train LCNMT, we automatically extract constraints from parallel corpora. By combining word alignment (to find mappings) and syntactic parsing (to identify coherent phrases), we generate a high-quality set of phrase pairs. This pipeline allows the model to learn from diverse lexical constraints without requiring manually labeled data.

Experimental Results

We evaluated LCNMT on the WMT’17 Chinese-to-English (Zh-En) and English-to-German (En-De) tasks. We simulated real-world usage by providing varying ratios of constraints (30%, 50%, and 70%) during testing.

BLEU score comparison on the newstest_ende task, showing LCNMT outperforming the Base Transformer and DBA across all constraint ratios.

As illustrated in the results for the English-to-German task (Figure 5), LCNMT consistently outperforms both the baseline Transformer and the state-of-the-art Dynamic Beam Allocation (DBA) method. At a 70% constraint ratio, LCNMT achieves a BLEU score of 33.8, compared to 32.2 for DBA and 27.3 for the base model. Notably, LCNMT maintains the decoding speed of the base model, whereas DBA and similar methods incur significant computational overhead.

Conclusion

LCNMT provides a robust framework for lexical constraints in NMT. By integrating constraints into the encoder's representation through an external memory and memory-attention mechanism, the model achieves superior translation quality and fluency while maintaining high inference efficiency. This approach paves the way for more responsive and steerable machine translation systems in production environments.

Summary updated

Abstract

Recent years has witnessed dramatic progress of neural machine translation (NMT), however, the method of manually guiding the translation procedure remains to be better explored. Previous works proposed to handle such problem through lexcially-constrained beam search in the decoding phase. Unfortunately, these lexically-constrained beam search methods suffer two fatal disadvantages: high computational complexity and hard beam search which generates unexpected translations. In this paper, we propose to learn the ability of lexically-constrained translation with external memory, which can overcome the above mentioned disadvantages. For the training process, automatically extracted phrase pairs are extracted from alignment and sentence parsing, then further be encoded into an external memory. This memory is then used to provide lexically-constrained information for training through a memory-attention machanism. Various experiments are conducted on WMT Chinese to English and English to German tasks. All the results can demonstrate the effectiveness of our method.
BibTeX
@misc{li2019learning,
  author = {Ya Li and Xinyu Liu and Dan Liu and Xueqiang Zhang and Junhua Liu},
  title = {{Learning Efficient Lexically-Constrained Neural Machine Translation with External Memory}},
  year = {2019},
  eprint = {1901.11344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.11344},
}
Figure