Deep Recurrent Generative Decoder for Abstractive Text Summarization

Piji Li, Wai Lam, Lidong Bing, Zihao Wang

Abstract

Abstractive text summarization requires models to move beyond simple extraction and generate concise, coherent sentences that capture the essence of a source document. While standard sequence-to-sequence models have made significant strides, their deterministic nature often fails to capture the high-level latent structures inherent in human-written summaries. This paper introduces the Deep Recurrent Generative Decoder (DRGD), a framework that integrates recurrent latent variables into the decoding process. By combining deterministic hidden states with a stochastic latent structure model, DRGD effectively captures the "blueprints" of summaries, achieving state-of-the-art performance on major English and Chinese benchmarks.

Introduction

Abstractive summarization is often considered a "holy grail" of natural language processing. Unlike extractive methods that copy-paste existing sentences, abstractive systems must understand the source text and generate entirely new phrasing. A major limitation in existing neural models is their reliance on deterministic transformations. Human summaries often follow specific high-level structures—such as "Who did What to Whom"—which are difficult for fixed mathematical mappings to represent. While Variational Auto-Encoders (VAEs) have been explored to capture these latent structures, they typically lack the recurrent dependencies necessary to track how a summary's structure should evolve from the beginning of a sentence to the end.

Methodology: The DRGD Framework

The core innovation of this work is the Deep Recurrent Generative Decoder (DRGD). The model bifurcates the decoding process into two parallel, interacting tracks:

  1. The Deterministic Track: Utilizing a standard Gated Recurrent Unit (GRU) with an attention mechanism, this track manages the "local" logic of predicting the next word based on the previous token and the source context.
  2. The Recurrent Generative Track: This track incorporates a latent random variable ($z_t$) at every time step. Crucially, $z_t$ depends on the previous latent variable $z_{t-1}$, allowing the model to maintain a "structural memory" or a consistent plan throughout the generation process.
The DRGD architecture showing the interaction between the deterministic encoder-decoder and the recurrent latent variables.

Training this system involves Neural Variational Inference. Because the true posterior distribution of the latent variables is mathematically intractable, the authors use a recognition model to approximate it, optimizing the Variational Lower Bound (ELBO). The final word generation is derived from a hybrid hidden state ($h^{dy}_t$) that synthesizes information from both the deterministic and stochastic tracks.

Experimental Setup and Datasets

The DRGD model was evaluated on three diverse benchmarks to demonstrate its cross-lingual effectiveness:

  • Gigawords (English): 3.8 million sentence-headline pairs for short-form summarization.
  • DUC-2004 (English): 500 documents with expert-written summaries for standard evaluation.
  • LCSTS (Chinese): 2.4 million short-text pairs from Sina Weibo, testing the model on non-Latin scripts.

Key Results

DRGD consistently outperformed strong baselines, including models with specialized copying or distraction mechanisms. On the Gigawords dataset, DRGD achieved a ROUGE-1 score of 36.27, surpassing previous state-of-the-art models like lvt5k-1sent (35.30).

Comparison of DRGD against various baseline systems on the Gigawords dataset.

Qualitative analysis further supported these findings. While standard decoders often produced irrelevant or "loose" sentences—sometimes confusing subjects in sports reporting—DRGD more accurately captured the "Who Action What" structure found in human-written "golden" summaries. This suggests that the latent variables successfully act as a "blueprint" for sentence planning.

Discussion and Future Work

The success of DRGD highlights the importance of modeling high-level themes and syntactic features through stochastic variables. However, the authors note that the model's complexity is higher than standard seq2seq architectures. Future research directions include integrating a "copy" or "pointer" mechanism to better handle rare words and proper nouns, as well as exploring "coverage" mechanisms to further reduce redundancy in longer generated summaries.

Summary updated

Abstract

We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder (DRGN). Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality. Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables. Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states. Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-of-the-art methods.
BibTeX
@inproceedings{li2017deep,
  author = {Piji Li and Wai Lam and Lidong Bing and Zihao Wang},
  title = {{Deep Recurrent Generative Decoder for Abstractive Text Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1222},
}
Figure