Iterative Document Representation Learning Towards Summarization with Polishing

Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, Rui Yan

Introduction

Extractive summarization models typically operate in a single linear pass, selecting sentences based on a one-time document representation. However, this approach differs significantly from human cognition. When humans summarize complex texts, we often engage in an iterative process: skimming for the "gist," re-reading specific sections to clarify details, and refining our understanding before finalizing the most salient points.

This paper introduces Iterative Text Summarization (ITS), a novel framework that mimics this "polishing" behavior. By iteratively updating document representations, the model achieves a more nuanced understanding of text, leading to state-of-the-art performance in extractive summarization.

Architecture: The Iterative Loop

The ITS architecture is designed to treat document understanding as an evolving process rather than a static computation. It consists of three primary components that work in a feedback loop.

The ITS architecture featuring the Selective-reading module, Iterative Unit, and multi-pass Decoder structure.

1. Selective Reading Module

Unlike standard recurrent units, the Selective Reading Module decides how much to update a sentence's representation by comparing it to the current global document representation ($D_{k-1}$). If a sentence aligns closely with the "polished" gist of the document from previous passes, the model allocates more attention to it. This mechanism prevents the model from being distracted by redundant or irrelevant details.

2. The Iterative Unit

After completing a full pass through the document's sentences, the Iterative Unit takes the final hidden state and updates the global document representation ($D_k$). This step represents the "polishing" phase, where the model integrates new insights into its overall understanding of the text.

3. Sentence Labeling Module

To make the final selection, the model does not rely solely on the final iteration. Instead, it concatenates hidden states from all iterations. This allows the classifier to observe how a sentence's perceived importance evolved over multiple passes, providing a more robust signal for the final "keep or toss" decision.

The Impact of Iteration

A key finding of this research is the "sharpening" effect of multiple passes. In early iterations, the model's confidence is often distributed across many sentences. As iterations progress, the probability scores for salient sentences become more distinct.

ROUGE scores across different iteration numbers, showing peak performance around the fifth pass.

As shown in the performance metrics, there is a clear "sweet spot" for iteration. Performance on the CNN/DailyMail dataset improves steadily until approximately the fifth iteration, after which it plateaus or slightly declines. This suggests that while "reading twice" (or five times) is beneficial, excessive polishing can lead to overfitting or the inclusion of marginal details.

Experimental Results

The ITS model was rigorously evaluated against several benchmarks:

  • CNN/DailyMail: ITS achieved a ROUGE-1 score of 27.4, outperforming established baselines such as SummaRuNNer (26.2) and Hybrid MemNet (26.3).
  • DUC2002: To test generalization, the model was evaluated on this out-of-domain dataset without retraining. It maintained a leading ROUGE-1 score of 47.6, demonstrating that the iterative logic is robust across different writing styles.
  • Human Evaluation: In blind rankings, human judges consistently rated ITS summaries as more informative and coherent than those produced by other AI systems, placing it second only to human-written gold standards.

Conclusion and Future Directions

The ITS framework demonstrates that iterative refinement is a powerful tool for document understanding. By allowing a model to "re-read" and "polish" its internal representations, we can bridge the gap between simple linear processing and complex human-like summarization.

While the current model is more computationally intensive due to its multi-pass nature, the gains in summary quality are significant. Future work may explore applying this iterative logic to abstractive summarization, where the ability to refine a "draft" understanding could lead to even more coherent and less repetitive generated text.

Summary updated

BibTeX
@inproceedings{chen2018iterative,
  author = {Xiuying Chen and Shen Gao and Chongyang Tao and Yan Song and Dongyan Zhao and Rui Yan},
  title = {{Iterative Document Representation Learning Towards Summarization with Polishing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1442},
}
Figure