A Survey on Neural Network-Based Summarization Methods

Yue DongarXiv 2018 · OpenReview

Abstract: Automatic text summarization, the automated process of shortening a text while reserving the main ideas of the document(s), is a critical research area in natural language processing. The aim of this literature review is to survey the recent work on neural-based models in automatic text summarization. We examine in detail ten state-of-the-art neural-based summarizers: five abstractive models and five extractive models. In addition, we discuss the related techniques that can be applied to the summarization tasks and present promising paths for future research in neural-based summarization.

Introduction

In an era of information overload, the "TL;DR" (Too Long; Didn't Read) is no longer a luxury—it’s a necessity. Automatic text summarization aims to distill thousands of words into a few salient sentences. This survey tracks the evolution of the field from 2014 to 2018, marking the transition from unsupervised information retrieval to sophisticated neural encoder-decoder models. It categorizes the field into extractive (selecting existing sentences) and abstractive (generating new text) methods, highlighting how innovations like attention mechanisms and reinforcement learning have redefined the state of the art.

Problem Definition

The core challenge of automatic text summarization is shortening a text while preserving its main ideas. Historically, this was achieved using unsupervised models that ranked sentences based on word frequency or graph-based importance (e.g., TextRank). However, traditional methods struggled with nuance and were limited to a "copy-paste" approach. The emergence of deep learning has allowed neural networks to learn better internal representations of meaning and generate more human-like, cohesive summaries.

Extractive Summarization Models

Extractive models treat summarization as a classification or ranking task, selecting existing sentences from the source text. Early neural approaches utilized Recursive Auto-encoders (RAE) or CNNs to create sentence embeddings, followed by optimization algorithms to select a diverse and representative set. Later models, such as SummaRuNNer, employed bi-directional RNNs to evaluate sentences in context, making selection decisions based on content, position, and novelty relative to the summary generated thus far.

Comparison of neural extractive summarization models (2014-2017).

Abstractive Summarization Models

Abstractive models are more complex, typically utilizing an Encoder-Decoder (Sequence-to-Sequence) framework to generate new text. Key innovations in this area include:

  • Attention Mechanisms: These allow the decoder to focus on specific parts of the source document while generating each word.
  • Pointer-Generator Networks: A hybrid approach where the model can either generate a word from its vocabulary or "point" to and copy a rare word (like a proper noun) from the source text.
  • Coverage Mechanisms: Designed to solve the "repetition problem" where models loop on the same phrase.
  • Reinforcement Learning (RL): Models like the Neural Intra-attention Model use RL to optimize directly for ROUGE scores rather than just next-word prediction.

Datasets and Evaluation

The field relies on several "gold standard" benchmarks:

  • DUC2002: A classic dataset for extractive evaluation.
  • Gigaword: Used for headline generation (short, one-sentence abstractive summaries).
  • CNN/DailyMail: The primary dataset for multi-sentence summarization, containing ~300k article-highlight pairs.

Performance is typically measured using ROUGE scores (n-gram overlap). For extractive tasks on DUC2002, the CNNLM (2015) model achieved a ROUGE-1 score of 51.0. For abstractive tasks on CNN/DailyMail, the Neural Intra-attention Model (2017) reached a ROUGE-L score of 36.90.

Key Technical Innovations

  • The 2014 Pivot: Neural-based continuous vector models began outperforming traditional IR techniques in 2014.
  • Submodularity: Effective extractive summaries must balance coverage with redundancy, often modeled via submodular functions.
  • The OOV Problem: The Pointer-Generator concept addressed the challenge of "Out-of-Vocabulary" words.
  • Exposure Bias: RL-based training helps bridge the gap between training (where the model sees ground truth) and testing (where it relies on its own previous predictions).

Limitations and Future Directions

Despite significant progress, several bottlenecks remain:

  1. Memory Constraints: Processing very long documents is difficult due to memory limits.
  2. Transparency: Neural models remain "black boxes" regarding their selection logic.
  3. Domain Specificity: There is a lack of large-scale parallel data for specialized domains like legal or medical text.

Future research is likely to focus on Text Simplification and more advanced Actor-Critic RL algorithms to improve readability and accuracy.

Summary updated

Abstract

Automatic text summarization, the automated process of shortening a text while reserving the main ideas of the document(s), is a critical research area in natural language processing. The aim of this literature review is to survey the recent work on neural-based models in automatic text summarization. We examine in detail ten state-of-the-art neural-based summarizers: five abstractive models and five extractive models. In addition, we discuss the related techniques that can be applied to the summarization tasks and present promising paths for future research in neural-based summarization.
BibTeX
@misc{dong2018survey,
  author = {Yue Dong},
  title = {{A Survey on Neural Network-Based Summarization Methods}},
  year = {2018},
  eprint = {1804.04589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.04589},
}
Figure