Attention With Sparsity Regularization for Neural Machine Translation and Summarization
Lists
Only you see this
Abstract
The attention mechanism has become the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de facto</i> standard component in neural sequence to sequence tasks, such as machine translation and abstractive summarization. It dynamically determines which parts in the input sentence should be focused on when generating each word in the output sequence. Ideally, only few relevant input words should be attended to at each decoding time step and the attention weight distribution should be sparse and sharp. However, previous methods have no good mechanism to control this attention weight distribution. In this paper, we propose a sparse attention model in which a sparsity regularization term is designed to augment the objective function. We explore two kinds of regularizations: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{\infty }$</tex-math></inline-formula> -norm regularization and minimum entropy regularization, both of which aim to sharpen the attention weight distribution. Extensive experiments on both neural machine translation and abstractive summarization demonstrate that our proposed sparse attention model can substantially outperform the strong baselines. And the detailed analyses reveal that the final attention distribution indeed becomes sparse and sharp.
BibTeX
@article{zhang2019attention,
author = {Jiajun Zhang and Yang Zhao and Haoran Li and Chengqing Zong},
title = {{Attention With Sparsity Regularization for Neural Machine Translation and Summarization}},
journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
year = {2019},
doi = {10.1109/taslp.2018.2883740},
}Loading related content...