Decoupling Encoder and Decoder Networks for Abstractive Document Summarization

Ying Xu, Jey Han Lau, Timothy Baldwin, Trevor Cohn

Introduction

Abstractive summarization represents a significant challenge in natural language processing, requiring models to generate novel sentences that encapsulate the core meaning of a source document. While state-of-the-art approaches primarily utilize attentive encoder-decoder architectures, these models are computationally expensive and data-intensive. They typically require joint training of both components over millions of examples, often taking several days to converge. This paper investigates whether the encoding and decoding tasks can be effectively decoupled to improve efficiency without sacrificing performance.

The Recurrent Decoupled Summarizer (RDS)

The authors propose the Recurrent Decoupled Summarizer (RDS), an architecture that separates the document encoder from the summary decoder. Unlike traditional models where the encoder is trained alongside the decoder, the RDS utilizes a pre-trained, unsupervised document encoder.

The RDS architecture decouples the encoder (doc2vec) from the RNN decoder.

Architecture Components

The RDS framework consists of two primary modules:

  • Encoder: The system employs doc2vec (specifically the Distributed Bag of Words variant) to generate a fixed-length vector representation of the entire document. This encoder is pre-trained on a large corpus, and its weights remain static during the training of the summarization task.
  • Decoder: A Recurrent Neural Network (RNN), specifically an LSTM, serves as the decoder. It functions as a conditional language model, using the fixed document vector as a global signal to guide the generation of the summary.

The authors evaluated six different methods for integrating the document vector into the decoder, finding that "stacking" the vector with the input at each time step provided the most consistent results. Notably, the RDS lacks an explicit attention mechanism. The decoder relies solely on the "gist" vector provided by the encoder, which forces the model to generate summaries based on a global representation rather than word-for-word copying from the source.

Experimental Results

The model was evaluated on the GIGAWORD dataset (~4.3 million news articles) for in-domain testing and the DUC03/04 datasets for out-of-domain generalization.

In-Domain Performance and Efficiency

On the GIGAWORD task, the RDS model demonstrated competitive performance. It achieved a ROUGE-1 score of 30.7, surpassing basic joint models like ABS and approaching the performance of more complex attentive architectures like RAS-Elman.

ROUGE scores on the GIGAWORD dataset highlight the competitive in-domain performance of the RDS model.

Beyond accuracy, the decoupled approach offered significant efficiency gains. The RDS model reached competitive performance in just 2 days of training, compared to 4 days for standard coupled baselines. Furthermore, the model exhibited a much lower "source copy rate" (~50% vs ~90% for traditional models), indicating a higher degree of true abstraction and paraphrasing. The results also show that the model remains robust even when trained on significantly less data (75% or 50% of the original set).

Generalization Challenges

While successful in-domain, the RDS model struggled with out-of-domain data. On the DUC benchmarks, its performance dropped significantly (16.7 ROUGE-1 compared to a 22.4 baseline for Prefix). This suggests that the fixed, unsupervised document representations may not capture the nuances required for different writing styles or topics encountered outside the training distribution.

Conclusion

The RDS architecture demonstrates that decoupling encoding and decoding can significantly reduce training time and parameter counts while maintaining strong in-domain performance. By removing the attention mechanism, the model is forced to perform genuine abstraction. However, the reliance on a fixed, unsupervised document signal currently limits the model's ability to generalize across diverse domains. Future research will focus on enhancing cross-domain robustness while preserving the efficiency of the decoupled framework.

Summary updated

Abstract

Abstractive document summarization seeks to automatically generate a summary for a document, based on some abstract “understanding” of the original document. State-of-the-art techniques traditionally use attentive encoder–decoder architectures. However, due to the large number of parameters in these models, they require large training datasets and long training times. In this paper, we propose decoupling the encoder and decoder networks, and training them separately. We encode documents using an unsupervised document encoder, and then feed the document vector to a recurrent neural network decoder. With this decoupled architecture, we decrease the number of parameters in the decoder substantially, and shorten its training time. Experiments show that the decoupled model achieves comparable performance with state-of-the-art models for in-domain documents, but less well for out-of-domain documents.
BibTeX
@inproceedings{xu2017decoupling,
  author = {Ying Xu and Jey Han Lau and Timothy Baldwin and Trevor Cohn},
  title = {{Decoupling Encoder and Decoder Networks for Abstractive Document Summarization}},
  booktitle = {MultiLing@EACL},
  year = {2017},
  doi = {10.18653/v1/w17-1002},
}
Figure