Abstract
Recent years have witnessed dramatic progress in neural machine translation (NMT); however, methods for manually guiding the translation procedure remain to be better explored. Previous works proposed handling such problems through lexically-constrained beam search in the decoding phase. Unfortunately, these methods suffer from high computational complexity and "hard" beam search constraints that can generate unexpected or disfluent translations. We propose to learn lexically-constrained translation using an external memory, overcoming these disadvantages. Our model, LCNMT, encodes automatically extracted phrase pairs into an external memory, which provides constrained information during training via a memory-attention mechanism. Experiments on WMT Chinese-to-English and English-to-German tasks demonstrate the effectiveness of our approach.
Introduction
Standard NMT models often struggle with specific terminology requirements, such as technical jargon or brand names. While lexically-constrained decoding (e.g., Grid Beam Search or Dynamic Beam Allocation) forces these terms into the output, it significantly increases inference latency and often compromises sentence fluency. This paper introduces Lexically-Constrained Neural Machine Translation (LCNMT), which shifts the constraint logic from the decoding stage to the model architecture itself, allowing for efficient, high-quality translations.
The LCNMT Architecture
The core of LCNMT is a memory-augmented Transformer encoder. Instead of forcing constraints during decoding, we provide them as "suggestions" to the encoder, allowing the model to integrate them naturally into the context representation.
As shown in Figure 1, the architecture includes:
- External Memory: Stores constraint pairs (source phrase $\rightarrow$ target phrase) as Key-Value pairs.
- Memory-Attention Layer: A specialized layer in the encoder that allows the model to "look up" relevant target-side terminology based on the source context.
- Attention Loss: A supervised loss function that trains the model to attend to the correct memory slots, ensuring the constraints are effectively utilized.
Automatic Constraint Extraction
To train LCNMT, we automatically extract constraints from parallel corpora. By combining word alignment (to find mappings) and syntactic parsing (to identify coherent phrases), we generate a high-quality set of phrase pairs. This pipeline allows the model to learn from diverse lexical constraints without requiring manually labeled data.
Experimental Results
We evaluated LCNMT on the WMT’17 Chinese-to-English (Zh-En) and English-to-German (En-De) tasks. We simulated real-world usage by providing varying ratios of constraints (30%, 50%, and 70%) during testing.
As illustrated in the results for the English-to-German task (Figure 5), LCNMT consistently outperforms both the baseline Transformer and the state-of-the-art Dynamic Beam Allocation (DBA) method. At a 70% constraint ratio, LCNMT achieves a BLEU score of 33.8, compared to 32.2 for DBA and 27.3 for the base model. Notably, LCNMT maintains the decoding speed of the base model, whereas DBA and similar methods incur significant computational overhead.
Conclusion
LCNMT provides a robust framework for lexical constraints in NMT. By integrating constraints into the encoder's representation through an external memory and memory-attention mechanism, the model achieves superior translation quality and fluency while maintaining high inference efficiency. This approach paves the way for more responsive and steerable machine translation systems in production environments.
Summary updated