Auto-regressive extractive summarization with replacement

Tianyu Zhu, Wen Hua, Jianfeng Qu, Saeid Hosseini, Xiaofang Zhou

Introduction

Extractive summarization aims to produce a concise summary by selecting a subset of salient sentences from a source document. While effective, traditional auto-regressive models often suffer from "lead bias"—a tendency to over-prioritize early sentences—and "partial extraction discrepancy," where the internal representation of the summary becomes cluttered with information from sentences the model ultimately decided to ignore. This paper presents AES-Rep (Auto-regressive Extractive Summarization with Replacement), a novel framework that introduces a replacement mechanism to dynamically update the summary, mitigating these long-standing issues.

The Limitations of Sequential Extraction

Most existing extractive models process documents sequentially, making binary "keep" or "ignore" decisions for each sentence. This approach has two primary drawbacks. First, the "running memory" or partial summary representation is often constructed as a weighted average of all processed sentences. As shown in Figure 1, this includes sentences with low extraction probabilities, introducing noise that degrades the quality of subsequent decisions.

Standard auto-regressive models often incorporate noise from ignored sentences into the partial summary representation.

Second, sequential models are prone to lead bias. In news articles, critical information is typically concentrated at the beginning. Once a model fills its summary quota with these early sentences, it lacks the flexibility to replace them if a more concise or comprehensive sentence appears later in the text.

The AES-Rep Framework

AES-Rep addresses these challenges by expanding the model's action space. In addition to "Ignore" and "Add," the model gains a third capability: "Replace."

Architecture and Mechanism

The architecture of AES-Rep, illustrated in Figure 2, consists of several key components:

  1. Graph-Based Encoding: The model utilizes a heterogeneous graph where words and sentences are nodes. Graph Attention Networks (GAT) allow for global context exchange, ensuring that sentence representations are informed by the entire document context.
  2. Clean Summary Representation: Unlike previous models, AES-Rep maintains a "clean" representation of the partial summary, excluding noise from ignored sentences. This prevents error accumulation during the extraction process.
  3. Extraction and Replacement Modules: The Extraction Classifier determines if a sentence is salient. If a sentence is deemed important but redundant with existing content, the Replacement Locater identifies which previously selected sentence should be evicted to make room for the new, superior one.
The AES-Rep architecture enables the model to add or replace sentences based on a clean summary representation.

To further refine the replacement logic, the authors incorporated "distance embeddings." Since redundant sentences often appear in close proximity, this feature helps the model recognize optimal swap candidates. Additionally, an "Attentive Loss" function guided by ROUGE distributions ensures the model focuses on sentences with the highest information density.

Experimental Evaluation

The model was evaluated on the CNN/DailyMail and WikiHow datasets. On the CNN/DailyMail benchmark, AES-Rep achieved a ROUGE-1 score of 43.21, outperforming state-of-the-art non-BERT models like HSG+Tri-Blocking (42.95) and significantly beating older auto-regressive models like SummaRuNNer (39.60). Ablation studies confirmed that the "Replace" functionality is critical; removing it led to a significant drop in performance across all metrics.

In out-of-domain testing on WikiHow, AES-Rep demonstrated robustness, although it faced challenges against abstractive models on the ROUGE-2 metric. This suggests that while replacement improves extractive quality, some document types still benefit from the paraphrasing capabilities of abstractive systems.

Conclusion

AES-Rep represents a significant advancement in extractive summarization by rethinking the sequential decision-making process. By allowing the model to "change its mind" through a replacement mechanism and maintaining a noise-free summary representation, AES-Rep effectively mitigates lead bias and improves summary coherence. Future work may involve integrating this replacement logic into abstractive frameworks to enable real-time editing of generated text.

Summary updated

Abstract

Auto-regressive extractive summarization approaches determine sentence extraction probability conditioning on previous decisions by maintaining a partial summary representation. Despite its popularity, the framework has two main drawbacks: 1) the partial summary representation is irresolutely denoted by a weighted summation of all the processed sentences without any filtering, resulting in a noisy representation and degrading the effectiveness of extracting subsequent sentences; 2) earlier sentences are biased towards a higher extraction probability due to the sequential nature of sequence tagging. To address these two problems, we propose the Auto-regressive Extractive Summarization with Replacement (AES-Rep), a novel auto-regressive extractive summarization model. In particular, the AES-Rep model consists of two main modules: the extraction decision module that determines whether a sentence should be extracted, and the replacement locater module that enables extracted deficient sentences to be replaced with latter sentences by comparing their expressiveness with respect to the main idea of the document. These modules update the partial summary with explicit actions using elaborated multidimensional guidance. We conduct extensive experiments on the benchmark CNN and DailyMail datasets. Experimental results show that AES-Rep can achieve better performance compared with various strong baselines in terms of multiple ROUGE metrics.
BibTeX
@article{zhu2023auto,
  author = {Tianyu Zhu and Wen Hua and Jianfeng Qu and Saeid Hosseini and Xiaofang Zhou},
  title = {{Auto-regressive extractive summarization with replacement}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-022-01108-0},
}
Figure