Iterative Document Representation Learning Towards Summarization with Polishing

Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, Rui Yan

Introduction

Extractive summarization aims to identify and select the most salient sentences from a document to create a concise summary. Traditional state-of-the-art models typically process documents in a single pass, reading from start to finish before making a final selection. However, this "one-and-done" approach contrasts sharply with human cognitive processes. When summarizing complex texts, humans often browse for a general gist, then re-read specific sections to refine their understanding. This paper introduces Iterative Text Summarization (ITS), a framework designed to mimic this reflective behavior by polishing document representations through multiple reading passes.

The ITS Framework

The core of the ITS model is its ability to iteratively refine its internal understanding of a document. The architecture consists of three primary components that operate in a loop:

  1. Selective Reading Module: Unlike standard Gated Recurrent Units (GRUs), this module incorporates a specialized gating mechanism. It takes the current sentence and the global document representation from the previous iteration as inputs, deciding which information to update or retain based on its relevance to the evolving "gist."
  2. Iterative Unit: This component updates the global document representation ($D_k$) after each pass, effectively "refreshing" the model's context.
  3. Sentence Labeling Module: To make final predictions, the model concatenates hidden states from all iterations. A multi-layer perceptron (MLP) analyzes this "history of understanding" to assign importance scores.
The ITS architecture showing the iterative flow between the Encoder, Iterative Unit, and Decoder modules.

Experimental Results

The ITS model was evaluated on two benchmark datasets: CNN/DailyMail for large-scale supervised learning and DUC2002 for testing cross-domain generalization.

Quantitatively, ITS consistently outperformed previous state-of-the-art extractive models such as SummaRuNNer and Hybrid MemNet. On the CNN/DailyMail dataset, it achieved a Rouge-1 score of 27.4 (at 75 bytes), representing a significant improvement over the 26.2–26.3 range of its predecessors. Furthermore, the model maintained its performance lead on DUC2002, suggesting that the iterative mechanism learns robust reading strategies rather than just memorizing dataset-specific patterns.

Analysis: The Power of Polishing

A key insight of this work is the "polishing" effect observed over multiple iterations. The authors found that performance improves steadily up to approximately five iterations before beginning to decline, suggesting a "sweet spot" where the model has sufficiently filtered noise without over-analyzing.

Heatmap visualization showing how sentence importance probabilities evolve and saturate over multiple iterations.

As shown in the heatmaps, early iterations often assign similar probabilities to many sentences. As the process continues, the selective reading module acts as a contextual filter. By comparing sentences to the previous iteration's summary, the model identifies redundant information and focuses on truly salient content, leading to a "saturation" of focus on the best summary candidates.

Conclusion

The ITS framework demonstrates that "reading twice" significantly enhances an AI's ability to extract meaningful summaries. While the iterative approach introduces some computational overhead, the gains in summary quality and generalization are substantial. Future work may explore applying this iterative logic to abstractive summarization or implementing dynamic "early exit" strategies to optimize computational efficiency.

Summary updated

Abstract

In this paper, we introduce Iterative Text Summarization (ITS), an iteration-based model for supervised extractive text summarization, inspired by the observation that it is often necessary for a human to read an article multiple times in order to fully understand and summarize its contents. Current summarization approaches read through a document only once to generate a document representation, resulting in a sub-optimal representation. To address this issue we introduce a model which iteratively polishes the document representation on many passes through the document. As part of our model, we also introduce a selective reading mechanism that decides more accurately the extent to which each sentence in the model should be updated. Experimental results on the CNN/DailyMail and DUC2002 datasets demonstrate that our model significantly outperforms state-of-the-art extractive systems when evaluated by machines and by humans.
BibTeX
@inproceedings{chen2018iterative,
  author = {Xiuying Chen and Shen Gao and Chongyang Tao and Yan Song and Dongyan Zhao and Rui Yan},
  title = {{Iterative Document Representation Learning Towards Summarization with Polishing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1442},
}
Figure