Attention-aware Inference: Generating Summarization Texts According to Attention Distributions

Ye Ma, Zixun Lan, Lu ZongarXiv 2020 · arXiv

Abstract

This paper proposes a novel inference algorithm for seq-to-seq models. It corrects beam search step-by-step via the optimal attention distribution to make the generated text attend to source tokens in a controlled way. Experiments show the proposed attention-aware inference produces summaries rather differently from the beam search, and achieves promising improvements of higher scores and greater conciseness. The algorithm is also proven robust as it remains to outperform beam search significantly even with corrupted attention distributions.

Introduction

In sequence-to-sequence (seq-to-seq) models used for summarization, beam search is the standard way to generate text. However, it has a few well-known "pitfalls." First, it often suffers from the "beam size curse," where increasing the beam size actually leads to lower-quality, generic results. Second, models often struggle with "coverage"—either repeating themselves or failing to attend to critical parts of the source text.

Previous attempts to fix this, like the "coverage penalty" in Google’s Neural Machine Translation (GNMT), were relatively crude. They assumed that every source token should receive an equal (uniform) amount of attention and usually only applied a penalty after a sentence was finished. In reality, some words are more important than others, and waiting until the end of a sentence is too late to fix a "hallucinating" or repetitive model. This paper introduces a novel inference algorithm that "corrects" beam search step-by-step by following a predicted optimal attention distribution.

Methodology

The authors propose Attention-aware Inference, which integrates a predictive component into the standard generation pipeline to guide the model on exactly how much focus each source token should receive.

Architecture of the Attention-aware Inference model showing the interaction between the Attention-Prediction Model and the BART framework.

The system consists of three main components:

  1. The Attention Prediction Model: Before generating a summary, a lightweight 2-layer Transformer encoder analyzes the source document to predict an "optimal attention distribution." This tells the model exactly how much total attention (weight) each word in the source text should receive throughout the entire summary.
  2. Step-by-Step Scoring: During beam search, the algorithm calculates an Attention Score at every single step. If a candidate word causes the cumulative attention on a source token to exceed its predicted "budget," that beam is penalized.
  3. Length Reward Mechanism: To prevent the model from simply cutting off early to avoid attention penalties, the authors designed a step-by-step length reward that encourages the model to hit a target length derived from the sum of the predicted attention.

Experimental Setup

The researchers evaluated their method using three major summarization benchmarks:

  • CNN/DM: A "lowly-abstractive" dataset where summaries are often similar to source sentences.
  • XSUM: A "highly-abstractive" dataset with very short, one-sentence summaries.
  • NEWSRoom: Used to test the robustness of the model on a dataset it wasn't specifically fine-tuned for.

Results and Discussion

The proposed method achieved significant improvements across multiple metrics. On CNN/DM, the method achieved a new state-of-the-art (at the time of writing) with ROUGE scores of R-1: 45.13, R-2: 21.77, and R-L: 42.04.

Performance comparison of Attention-aware Inference against standard BART and PEGASUS baselines on CNN/DM and XSUM datasets.

Beyond ROUGE scores, several key insights emerged:

  • Breaking the Beam Size Curse: Unlike standard beam search, where ROUGE scores often plateau or drop as beam size increases, Attention-aware inference actually improves with larger beams. This suggests the "curse" is caused by a lack of guidance, not a fundamental flaw in searching.
  • Conciseness and Efficiency: The summaries were significantly more concise. On CNN/DM, the average token length dropped from 82.3 to 64.9, bringing it much closer to the human reference length of 67.5. Despite the extra prediction step, the total inference time was lower than standard beam search because the generated summaries were shorter.
  • Creative Summaries: The method encourages the model to use more "novel" words (words not found in the source text). In CNN/DM tests, the percentage of novel words increased from 4.5% to 6.7%.
  • Robustness: Even when the predicted attention distribution was "corrupted" or taken from a different dataset, the algorithm still tended to outperform standard beam search.

Conclusion

Attention-aware inference provides a robust framework for controlling the generation process in seq-to-seq models. By treating attention as a finite "budget" to be managed, the model generates more concise, creative, and accurate summaries. Future work may extend this approach to other modalities, such as image captioning or data-to-text tasks.

Summary updated

Abstract

This paper proposes a novel inference algorithm for seq-to-seq models. It corrects beam search step-by-step via the optimal attention distribution to make the generated text attend to source tokens in a controlled way. Experiments show the proposed attention-aware inference produces summaries rather differently from the beam search, and achieves promising improvements of higher scores and greater conciseness. The algorithm is also proven robust as it remains to outperform beam search significantly even with corrupted attention distributions.
BibTeX
@misc{ma2020attention,
  author = {Ye Ma and Zixun Lan and Lu Zong},
  title = {{Attention-aware Inference: Generating Summarization Texts According to Attention Distributions}},
  year = {2020},
  eprint = {2009.06891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.06891},
}
Figure