Abstract
Lexically constrained neural machine translation (NMT), which leverages pre-specified translation to constrain NMT, has practical significance in interactive translation and NMT domain adaption. Previous work either modify the decoding algorithm or train the model on augmented dataset. These methods suffer from either high computational overheads or low copying success rates. In this paper, we investigate Att-Input and Att-Output, two alignment-based constrained decoding methods. These two methods revise the target tokens during decoding based on word alignments derived from encoder-decoder attention weights. Our study shows that Att-Input translates better while Att-Output is more computationally efficient. Capitalizing on both strengths, we further propose EAM-Output by introducing an explicit alignment module (EAM) to a pretrained Transformer. It decodes similarly as EAM-Output, except using alignments derived from the EAM. We leverage the word alignments induced from Att-Input as labels and train the EAM while keeping the parameters of the Transformer frozen. Experiments on WMT16 De-En and WMT16 Ro-En show the effectiveness of our approaches on constrained NMT. In particular, the proposed EAM-Output method consistently outperforms previous approaches in translation quality, with light computational overheads over unconstrained baseline.
Introduction
In many real-world scenarios, machine translation must adhere to strict terminology, such as company-specific product names or user-provided corrections. This task, known as Lexically Constrained NMT, is challenging for modern Transformer models which lack the explicit word-to-word alignments found in older statistical systems. Previous solutions typically involve modifying decoding algorithms (which are computationally expensive) or data augmentation (which often fails to guarantee constraint inclusion). This paper explores a more efficient "middle way" using alignment-based decoding.
The Aligner-Editor Framework
The proposed framework introduces two modular components: an Aligner and an Editor. During the decoding process, the Aligner analyzes the model's internal attention weights to determine which source word is currently being translated. If the Aligner identifies a "constrained" source word, the Editor intervenes by overriding the model's output probabilities to force the generation of the pre-specified target word.
Alignment Strategies: Att-Input vs. Att-Output
The authors evaluate two primary methods for deriving these alignments:
- ATT-OUTPUT: This method extracts alignments from the top layers of the decoder. While fast, it is less accurate because it attempts to predict the alignment before the target word is fully formed in the decoder's state.
- ATT-INPUT: This method utilizes a "look-ahead" forward pass to see what the model would generate, providing much higher alignment accuracy. However, this extra pass doubles the computational cost.
Explicit Alignment Module (EAM)
To combine the accuracy of ATT-INPUT with the speed of ATT-OUTPUT, the authors introduce the Explicit Alignment Module (EAM). This lightweight module is added to a frozen, pre-trained Transformer. Using a knowledge distillation approach, the EAM is trained to mimic the high-quality alignments of the slower ATT-INPUT "teacher" model. The resulting EAM-OUTPUT method allows for high-quality, constrained translation without the need for a look-ahead pass during inference.
Key Results
Experiments on WMT16 German-English (De-En) and Romanian-English (Ro-En) benchmarks demonstrate the superiority of the EAM-OUTPUT approach:
- Translation Quality: EAM-OUTPUT consistently achieved the highest BLEU scores, reaching 37.6 BLEU on De-En, compared to the unconstrained baseline of 31.9.
- Copying Success Rate (CSR): The method successfully included constraints 93% to 97% of the time, significantly outperforming other alignment-based baselines.
- Efficiency: EAM-OUTPUT is nearly as fast as unconstrained translation and roughly 1.5x to 1.6x faster than popular methods like Vectorized Dynamic Beam Allocation (VDBA).
Conclusion
By introducing an explicit alignment module, this research provides a practical, high-performance solution for enforcing lexical constraints in NMT. The EAM-OUTPUT method offers a plug-and-play solution that maintains the speed of the Transformer architecture while ensuring precise adherence to terminology constraints.
Summary updated