Towards Better Word Alignment in Transformer

Kai Song, Xiaoqing Zhou, Heng Yu, Zhongqiang Huang, Yue Zhang, Weihua Luo, Xiangyu Duan, Min Zhang

Abstract

While neural models based on the Transformer architecture achieve the State-of-the-Art translation performance, it is well known that the learned target-to-source attentions do not correlate well with word alignment. There is an increasing interest in inducing accurate word alignment in Transformer, due to its important role in practical applications such as dictionary-guided translation and interactive translation. In this article, we extend and improve the recent work on unsupervised learning of word alignment in Transformer on two dimensions: a) parameter initialization from a pre-trained cross-lingual language model to leverage large amounts of monolingual data for learning robust contextualized word representations, and b) regularization of the training objective to directly model characteristics of word alignments which results in favorable word alignments receiving more concentrated probabilities. Experiments on benchmark data sets of three language pairs show that the proposed methods can significantly reduce alignment error rate (AER) by at least 3.7 to 7.7 points on each language pair over two recent works on improving the Transformer's word alignment. Moreover, our methods can achieve better alignment results than GIZA++ on certain test sets.
BibTeX
@article{song2020towards,
  author = {Kai Song and Xiaoqing Zhou and Heng Yu and Zhongqiang Huang and Yue Zhang and Weihua Luo and Xiangyu Duan and Min Zhang},
  title = {{Towards Better Word Alignment in Transformer}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  doi = {10.1109/taslp.2020.2998278},
}

Loading related content...

Figure