When humans summarize a long document, we rarely read the entire text once and immediately write a perfect synthesis. Instead, we skim to find the important parts, highlight key sentences, and then weave those highlights into a coherent narrative.
In natural language processing, this "skim-then-write" intuition has crystallized into a dominant architectural pattern: the two-stage summarization pipeline. By decoupling the task of finding relevant information from the task of rewriting it, researchers have found a way to bypass the physical and cognitive limits of neural networks.
The Context Bottleneck
The fundamental problem in modern summarization is the "context window." Even as models grow more powerful, they struggle with extremely long inputs—such as scientific papers, meeting transcripts, or multi-document clusters. Processing these in one go is computationally expensive and often leads to "noise" that confuses the model.
Peter J Liu and colleagues at Google Brain illustrated this in Generating Wikipedia by Summarizing Long Sequences. They treated Wikipedia generation as a massive multi-document summarization task, arguing that the best way to handle thousands of source tokens is to first filter them. This "extractive-abstractive" approach allows a model to synthesize information that would otherwise exceed its context window by focusing only on the most relevant snippets.
The Extract then Abstract Paradigm
The two-stage pipeline typically consists of two distinct modules:
- The Extractor (The Filter): This stage identifies "salient spans" or "question-worthy" sentences. It acts as a high-bandwidth filter, discarding irrelevant fluff.
- The Abstractor (The Writer): This stage takes the filtered content and rewrites it into a fluent, concise summary.
Jiwei Tan and Xiaojun Wan pioneered this with a "coarse-to-fine" framework in From Neural Sentence Summarization to Headline Generation: A Coarse-to-Fine Approach. By first extracting important sentences using traditional techniques and then processing them through a neural network, they generated more accurate headlines than standard end-to-end models.
This modularity is particularly effective for noisy data. For instance, in The UCF Podcast Summarization System at TREC 2020, researchers used a "content selector" to pick the most important segments of a messy podcast transcript before passing them to a generative BART model, outperforming human-written descriptions.
Grounding and Factuality
Beyond just saving memory, two-stage pipelines serve as a powerful defense against "hallucinations"—the tendency of models to invent facts. When a model is forced to generate a summary based only on a set of extracted "evidence" sentences, it is much harder for it to drift away from the source material.
Kundan Krishna and Zachary C. Lipton demonstrated this in Generating SOAP Notes from Doctor-Patient Conversations. Their CLUSTER2SENT pipeline breaks the task into extracting evidence, clustering related points, and then generating sentences. This modularity significantly improved human-rated factuality compared to end-to-end models.
More recently, Md Rizwan Parvez introduced the Evidence to Generate (E2G) strategy in Evidence to Generate (E2G): A Single-agent Two-step Prompting for Context Grounded and Retrieval Augmented Reasoning. By forcing Large Language Models (LLMs) to explicitly extract rationales before generating a final answer, the system focuses the model's attention on high-signal information, mitigating the performance degradation usually seen in long documents.
The Challenge of Context Loss
While two-stage pipelines are efficient, they are not perfect. The biggest risk is "information loss." When an extractor pulls a sentence out of a document, it often loses the surrounding context (like what "it" refers to in the previous sentence).
Ruifeng Yuan addressed this in Preserve Context Information for Extract-Generate Long-Input Summarization Framework. Their CAEG framework uses "context prompts" to bridge the gap between isolated fragments. By prepending these prompts to the generator's input, they preserve the local and global dependencies that are often shattered during the extraction phase.
Why It Works: A Division of Labor
The success of the two-stage approach lies in a fundamental division of labor. Extraction is often a "locate" problem—finding the needle in the haystack—which can be handled by simpler, faster models or even non-neural ranking methods. Abstraction is a "synthesis" problem—understanding and rephrasing—which requires the heavy lifting of a Transformer.
As Yusen Zhang and Tao Yu found in An Exploratory Study on Long Dialogue Summarization: What Works and What's Next, a simple "retrieve-then-summarize" pipeline often outperforms even the most complex "long-context" transformers. By narrowing the focus of the generative model, we don't just make summarization faster; we make it more precise.