Introduction
Abstractive summarization represents a significant challenge in natural language processing, requiring models to generate novel sentences that encapsulate the core meaning of a source document. While state-of-the-art approaches primarily utilize attentive encoder-decoder architectures, these models are computationally expensive and data-intensive. They typically require joint training of both components over millions of examples, often taking several days to converge. This paper investigates whether the encoding and decoding tasks can be effectively decoupled to improve efficiency without sacrificing performance.
The Recurrent Decoupled Summarizer (RDS)
The authors propose the Recurrent Decoupled Summarizer (RDS), an architecture that separates the document encoder from the summary decoder. Unlike traditional models where the encoder is trained alongside the decoder, the RDS utilizes a pre-trained, unsupervised document encoder.
Architecture Components
The RDS framework consists of two primary modules:
- Encoder: The system employs
doc2vec(specifically the Distributed Bag of Words variant) to generate a fixed-length vector representation of the entire document. This encoder is pre-trained on a large corpus, and its weights remain static during the training of the summarization task. - Decoder: A Recurrent Neural Network (RNN), specifically an LSTM, serves as the decoder. It functions as a conditional language model, using the fixed document vector as a global signal to guide the generation of the summary.
The authors evaluated six different methods for integrating the document vector into the decoder, finding that "stacking" the vector with the input at each time step provided the most consistent results. Notably, the RDS lacks an explicit attention mechanism. The decoder relies solely on the "gist" vector provided by the encoder, which forces the model to generate summaries based on a global representation rather than word-for-word copying from the source.
Experimental Results
The model was evaluated on the GIGAWORD dataset (~4.3 million news articles) for in-domain testing and the DUC03/04 datasets for out-of-domain generalization.
In-Domain Performance and Efficiency
On the GIGAWORD task, the RDS model demonstrated competitive performance. It achieved a ROUGE-1 score of 30.7, surpassing basic joint models like ABS and approaching the performance of more complex attentive architectures like RAS-Elman.
Beyond accuracy, the decoupled approach offered significant efficiency gains. The RDS model reached competitive performance in just 2 days of training, compared to 4 days for standard coupled baselines. Furthermore, the model exhibited a much lower "source copy rate" (~50% vs ~90% for traditional models), indicating a higher degree of true abstraction and paraphrasing. The results also show that the model remains robust even when trained on significantly less data (75% or 50% of the original set).
Generalization Challenges
While successful in-domain, the RDS model struggled with out-of-domain data. On the DUC benchmarks, its performance dropped significantly (16.7 ROUGE-1 compared to a 22.4 baseline for Prefix). This suggests that the fixed, unsupervised document representations may not capture the nuances required for different writing styles or topics encountered outside the training distribution.
Conclusion
The RDS architecture demonstrates that decoupling encoding and decoding can significantly reduce training time and parameter counts while maintaining strong in-domain performance. By removing the attention mechanism, the model is forced to perform genuine abstraction. However, the reliance on a fixed, unsupervised document signal currently limits the model's ability to generalize across diverse domains. Future research will focus on enhancing cross-domain robustness while preserving the efficiency of the decoupled framework.
Summary updated