Lexically Constrained Neural Machine Translation with Explicit Alignment Guidance

Guanhua Chen, Yun Chen, Victor O. K. LiAAAI 2021 · OpenReview

Abstract

Lexically constrained neural machine translation (NMT), which leverages pre-specified translation to constrain NMT, has practical significance in interactive translation and NMT domain adaption. Previous work either modify the decoding algorithm or train the model on augmented dataset. These methods suffer from either high computational overheads or low copying success rates. In this paper, we investigate Att-Input and Att-Output, two alignment-based constrained decoding methods. These two methods revise the target tokens during decoding based on word alignments derived from encoder-decoder attention weights. Our study shows that Att-Input translates better while Att-Output is more computationally efficient. Capitalizing on both strengths, we further propose EAM-Output by introducing an explicit alignment module (EAM) to a pretrained Transformer. It decodes similarly as EAM-Output, except using alignments derived from the EAM. We leverage the word alignments induced from Att-Input as labels and train the EAM while keeping the parameters of the Transformer frozen. Experiments on WMT16 De-En and WMT16 Ro-En show the effectiveness of our approaches on constrained NMT. In particular, the proposed EAM-Output method consistently outperforms previous approaches in translation quality, with light computational overheads over unconstrained baseline.

Introduction

In many real-world scenarios, machine translation must adhere to strict terminology, such as company-specific product names or user-provided corrections. This task, known as Lexically Constrained NMT, is challenging for modern Transformer models which lack the explicit word-to-word alignments found in older statistical systems. Previous solutions typically involve modifying decoding algorithms (which are computationally expensive) or data augmentation (which often fails to guarantee constraint inclusion). This paper explores a more efficient "middle way" using alignment-based decoding.

The Aligner-Editor Framework

The proposed framework introduces two modular components: an Aligner and an Editor. During the decoding process, the Aligner analyzes the model's internal attention weights to determine which source word is currently being translated. If the Aligner identifies a "constrained" source word, the Editor intervenes by overriding the model's output probabilities to force the generation of the pre-specified target word.

The Aligner-Editor framework for lexically constrained decoding.

Alignment Strategies: Att-Input vs. Att-Output

The authors evaluate two primary methods for deriving these alignments:

  1. ATT-OUTPUT: This method extracts alignments from the top layers of the decoder. While fast, it is less accurate because it attempts to predict the alignment before the target word is fully formed in the decoder's state.
  2. ATT-INPUT: This method utilizes a "look-ahead" forward pass to see what the model would generate, providing much higher alignment accuracy. However, this extra pass doubles the computational cost.
Comparison between Att-Output and Att-Input alignment mechanisms.

Explicit Alignment Module (EAM)

To combine the accuracy of ATT-INPUT with the speed of ATT-OUTPUT, the authors introduce the Explicit Alignment Module (EAM). This lightweight module is added to a frozen, pre-trained Transformer. Using a knowledge distillation approach, the EAM is trained to mimic the high-quality alignments of the slower ATT-INPUT "teacher" model. The resulting EAM-OUTPUT method allows for high-quality, constrained translation without the need for a look-ahead pass during inference.

Key Results

Experiments on WMT16 German-English (De-En) and Romanian-English (Ro-En) benchmarks demonstrate the superiority of the EAM-OUTPUT approach:

  • Translation Quality: EAM-OUTPUT consistently achieved the highest BLEU scores, reaching 37.6 BLEU on De-En, compared to the unconstrained baseline of 31.9.
  • Copying Success Rate (CSR): The method successfully included constraints 93% to 97% of the time, significantly outperforming other alignment-based baselines.
  • Efficiency: EAM-OUTPUT is nearly as fast as unconstrained translation and roughly 1.5x to 1.6x faster than popular methods like Vectorized Dynamic Beam Allocation (VDBA).

Conclusion

By introducing an explicit alignment module, this research provides a practical, high-performance solution for enforcing lexical constraints in NMT. The EAM-OUTPUT method offers a plug-and-play solution that maintains the speed of the Transformer architecture while ensuring precise adherence to terminology constraints.

Summary updated

Abstract

Lexically constrained neural machine translation (NMT), which leverages pre-specified translation to constrain NMT, has practical significance in interactive translation and NMT domain adaption. Previous work either modify the decoding algorithm or train the model on augmented dataset. These methods suffer from either high computational overheads or low copying success rates. In this paper, we investigate Att-Input and Att-Output, two alignment-based constrained decoding methods. These two methods revise the target tokens during decoding based on word alignments derived from encoder-decoder attention weights. Our study shows that Att-Input translates better while Att-Output is more computationally efficient. Capitalizing on both strengths, we further propose EAM-Output by introducing an explicit alignment module (EAM) to a pretrained Transformer. It decodes similarly as EAM-Output, except using alignments derived from the EAM. We leverage the word alignments induced from Att-Input as labels and train the EAM while keeping the parameters of the Transformer frozen. Experiments on WMT16 De-En and WMT16 Ro-En show the effectiveness of our approaches on constrained NMT. In particular, the proposed EAM-Output method consistently outperforms previous approaches in translation quality, with light computational overheads over unconstrained baseline.
BibTeX
@inproceedings{chen2021lexically,
  author = {Guanhua Chen and Yun Chen and Victor O. K. Li},
  title = {{Lexically Constrained Neural Machine Translation with Explicit Alignment Guidance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=T9E9yQdWXXm},
}
Figure