Deep Recurrent Generative Decoder for Abstractive Text Summarization

Piji Li, Wai Lam, Lidong Bing, Zihao Wang

Introduction

Abstractive text summarization represents a significant challenge in natural language processing, requiring models to go beyond simple extraction to generate concise, coherent, and entirely new sentences that capture the essence of a source text. While sequence-to-sequence (seq2seq) models using Gated Recurrent Units (GRUs) or Long Short-Term Memory (LSTM) networks have become standard, they often struggle to replicate the high-level structural logic found in human-written summaries. This paper introduces the Deep Recurrent Generative Decoder (DRGD), a framework that enhances abstractive summarization by integrating a recurrent latent variable model into a standard deterministic RNN architecture.

The Challenge of Deterministic Summarization

Existing seq2seq models are primarily deterministic. While effective at local word prediction, they often lack the representational flexibility to model the "latent structures"—the underlying syntactic templates or topical flows—that humans naturally follow. For example, news headlines frequently adhere to a "Who (Subject) + Action (Verb) + What (Object)" structure. Purely deterministic models may generate grammatically correct text that nonetheless lacks structural coherence or focus because they cannot explicitly plan for these abstract patterns.

The Deep Recurrent Generative Decoder (DRGD)

The authors propose a hybrid architecture that merges discriminative and generative modeling to address these limitations. The system consists of a standard bidirectional GRU encoder with an attention mechanism, paired with the innovative Deep Recurrent Generative Decoder.

The DRGD architecture combining a bidirectional GRU encoder with a recurrent generative decoder.

Unlike standard Variational Auto-Encoders (VAEs) that utilize a single static latent vector, the DRGD employs a sequence of recurrent latent variables ($z_t$). At each time step, the latent state $z_t$ depends on the previous latent state ($z_{t-1}$) and the previously generated word ($y_{t-1}$). This allows the model to maintain a "stochastic plan" for the sentence structure as it unfolds. The final word prediction is a combination of the hidden state from a deterministic GRU (the discriminative part) and the latent variable (the generative part). To train this complex system, the authors utilize neural variational inference to optimize a lower bound on the marginal likelihood.

Experimental Setup and Datasets

The DRGD model was evaluated across three major benchmarks to demonstrate its cross-linguistic effectiveness:

  • Gigawords: A massive English dataset containing 3.8 million pairs, focused on generating headlines from the first sentence of news articles.
  • DUC-2004: A smaller, expert-curated English test set used to evaluate model generalization.
  • LCSTS: A large-scale Chinese short-text summarization dataset (2.4 million pairs) from Sina Weibo, characterized by informal and diverse language.

Results and Analysis

The DRGD model consistently outperformed existing state-of-the-art methods across ROUGE-1, ROUGE-2, and ROUGE-L metrics. On the Gigawords dataset, the model achieved a ROUGE-1 score of 36.27, surpassing previous benchmarks such as lvt5k-1sent (35.30).

Performance comparison on the Gigawords dataset.

Similar success was observed in the Chinese LCSTS dataset, where DRGD reached a ROUGE-1 score of 36.99, outperforming models like RNN-distract (35.20) and CopyNet (34.40). Qualitative case studies further supported these metrics; the DRGD model successfully captured "Who-Action-What" structures in instances where baseline models produced generic or irrelevant sentences. For example, in sports reporting, the DRGD correctly identified the winner and the specific event, whereas deterministic baselines often generated less accurate summaries.

Conclusion and Future Directions

By explicitly modeling the inherent structures of human summaries through recurrent latent variables, the DRGD framework provides a more flexible and accurate approach to abstractive summarization. While the model represents a significant step forward, future work could involve integrating "copying mechanisms" to better handle out-of-vocabulary words (such as specific names) and "coverage" mechanisms to prevent word repetition. Additionally, the computational overhead of variational inference remains a consideration for real-time applications, suggesting a need for further optimization in stochastic decoding processes.

Summary updated

BibTeX
@inproceedings{li2017deep,
  author = {Piji Li and Wai Lam and Lidong Bing and Zihao Wang},
  title = {{Deep Recurrent Generative Decoder for Abstractive Text Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1222},
}
Figure