Distraction-Based Neural Networks for Document Summarization

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui JiangarXiv 2016 · arXiv

Introduction

Standard attention mechanisms have revolutionized neural document summarization by allowing models to focus on salient parts of an input text. However, these mechanisms often suffer from a "tunnel vision" effect, particularly when processing long documents. Models tend to repeatedly attend to the same high-salience regions, leading to redundant summaries that lack global coverage. This paper proposes a "distraction" mechanism designed to force the model to traverse different parts of a document, mimicking the human cognitive process of scanning an entire text to ensure comprehensive summarization.

Methodology

The authors extend the standard Gated Recurrent Unit (GRU) encoder-decoder framework by introducing a distraction layer. This layer acts as a control mechanism to prevent the model from getting stuck on specific input regions.

Architecture of the distraction-based neural network model.

Three primary distraction variants were developed:

  1. M1 (Distraction over Content): This variant maintains a history of previously attended content. A penalty is applied to the current context based on this history, encouraging the model to seek novel information.
  2. M2 (Distraction over Attention Weights): Instead of penalizing content, this version tracks the history of attention weights. These historical weights serve as additional input for calculating new attention, signaling the model to explore unvisited regions.
  3. M3 (Distraction in Decoding): Implemented as a test-time constraint during beam search, this variant penalizes candidates that exhibit high similarity (measured via KL divergence or cosine similarity) to previous attention or hidden states.

Additionally, the model utilizes a two-layer hidden output to bridge the encoder and decoder more effectively and employs an unknown word replacement strategy to handle out-of-vocabulary terms by referencing the source document.

Experimental Setup

The proposed models were evaluated on two distinct datasets:

  • CNN News Collection: Comprising approximately 90,000 long-form articles (averaging ~775 words), this dataset serves as the primary benchmark for testing coverage and redundancy.
  • LCSTS (Large-scale Chinese Short Text Summarization): A massive dataset of ~2.4 million short microblogging pairs (~100 characters). This dataset tests the model's performance in scenarios where redundancy is less prevalent.

Results

The distraction-based models demonstrated significant improvements on long-form summarization tasks. On the CNN dataset, the M3 model achieved a ROUGE-1 score of 27.1, outperforming both basic Uni-GRU models and traditional extractive baselines like Lex-rank.

Performance comparison on the CNN dataset.

Key findings include:

  • Impact of Document Length: The benefit of distraction scales with document length. On a subset of CNN articles averaging 680 words, the model showed a 29% relative improvement over the baseline. On shorter articles (335 words), the improvement was 25.9%.
  • Short-Form Performance: On the LCSTS dataset, the distraction model matched the performance of a strong Bi-GRU baseline (35.2 ROUGE-1) but did not exceed it. This confirms that distraction is most effective when navigating long, complex documents where redundancy is a major challenge.

Conclusion

Distraction modeling provides a necessary counterweight to standard attention, ensuring that neural summarizers achieve better global coverage. While the mechanism adds complexity to the decoding process, its ability to significantly improve abstractive summarization for long documents makes it a valuable addition to the NLP toolkit. Future research may explore more expressive control layers and improved decoding strategies to further refine the transition from internal representations to generated text.

Summary updated

Abstract

Distributed representation learned with neural networks has recently shown to be effective in modeling natural languages at fine granularities such as words, phrases, and even sentences. Whether and how such an approach can be extended to help model larger spans of text, e.g., documents, is intriguing, and further investigation would still be desirable. This paper aims to enhance neural network models for such a purpose. A typical problem of document-level modeling is automatic summarization, which aims to model documents in order to generate summaries. In this paper, we propose neural models to train computers not just to pay attention to specific regions and content of input documents with attention models, but also distract them to traverse between different content of a document so as to better grasp the overall meaning for summarization. Without engineering any features, we train the models on two large datasets. The models achieve the state-of-the-art performance, and they significantly benefit from the distraction modeling, particularly when input documents are long.
BibTeX
@misc{chen2016distraction,
  author = {Qian Chen and Xiaodan Zhu and Zhen-Hua Ling and Si Wei and Hui Jiang},
  title = {{Distraction-Based Neural Networks for Document Summarization}},
  year = {2016},
  eprint = {1610.08462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.08462},
}
Figure