Introduction
Extractive summarization aims to produce a concise summary by selecting a subset of salient sentences from a source document. While effective, traditional auto-regressive models often suffer from "lead bias"—a tendency to over-prioritize early sentences—and "partial extraction discrepancy," where the internal representation of the summary becomes cluttered with information from sentences the model ultimately decided to ignore. This paper presents AES-Rep (Auto-regressive Extractive Summarization with Replacement), a novel framework that introduces a replacement mechanism to dynamically update the summary, mitigating these long-standing issues.
The Limitations of Sequential Extraction
Most existing extractive models process documents sequentially, making binary "keep" or "ignore" decisions for each sentence. This approach has two primary drawbacks. First, the "running memory" or partial summary representation is often constructed as a weighted average of all processed sentences. As shown in Figure 1, this includes sentences with low extraction probabilities, introducing noise that degrades the quality of subsequent decisions.
Second, sequential models are prone to lead bias. In news articles, critical information is typically concentrated at the beginning. Once a model fills its summary quota with these early sentences, it lacks the flexibility to replace them if a more concise or comprehensive sentence appears later in the text.
The AES-Rep Framework
AES-Rep addresses these challenges by expanding the model's action space. In addition to "Ignore" and "Add," the model gains a third capability: "Replace."
Architecture and Mechanism
The architecture of AES-Rep, illustrated in Figure 2, consists of several key components:
- Graph-Based Encoding: The model utilizes a heterogeneous graph where words and sentences are nodes. Graph Attention Networks (GAT) allow for global context exchange, ensuring that sentence representations are informed by the entire document context.
- Clean Summary Representation: Unlike previous models, AES-Rep maintains a "clean" representation of the partial summary, excluding noise from ignored sentences. This prevents error accumulation during the extraction process.
- Extraction and Replacement Modules: The Extraction Classifier determines if a sentence is salient. If a sentence is deemed important but redundant with existing content, the Replacement Locater identifies which previously selected sentence should be evicted to make room for the new, superior one.
To further refine the replacement logic, the authors incorporated "distance embeddings." Since redundant sentences often appear in close proximity, this feature helps the model recognize optimal swap candidates. Additionally, an "Attentive Loss" function guided by ROUGE distributions ensures the model focuses on sentences with the highest information density.
Experimental Evaluation
The model was evaluated on the CNN/DailyMail and WikiHow datasets. On the CNN/DailyMail benchmark, AES-Rep achieved a ROUGE-1 score of 43.21, outperforming state-of-the-art non-BERT models like HSG+Tri-Blocking (42.95) and significantly beating older auto-regressive models like SummaRuNNer (39.60). Ablation studies confirmed that the "Replace" functionality is critical; removing it led to a significant drop in performance across all metrics.
In out-of-domain testing on WikiHow, AES-Rep demonstrated robustness, although it faced challenges against abstractive models on the ROUGE-2 metric. This suggests that while replacement improves extractive quality, some document types still benefit from the paraphrasing capabilities of abstractive systems.
Conclusion
AES-Rep represents a significant advancement in extractive summarization by rethinking the sequential decision-making process. By allowing the model to "change its mind" through a replacement mechanism and maintaining a noise-free summary representation, AES-Rep effectively mitigates lead bias and improves summary coherence. Future work may involve integrating this replacement logic into abstractive frameworks to enable real-time editing of generated text.
Summary updated