Abstract
We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary. Our work addresses current limitations of models for abstractive summarization that often hallucinate information or generate summaries with coherence issues. To generate abstractive summaries with factual consistency and narrative flow, we propose Cooperative Generator -- Discriminator Networks (Co-opNet), a novel transformer-based framework where a generator works with a discriminator architecture to compose coherent long-form summaries. We explore four different discriminator objectives which each capture a different aspect of coherence, including whether salient spans of generated abstracts are hallucinated or appear in the input context, and the likelihood of sentence adjacency in generated abstracts. We measure the ability of Co-opNet to learn these objectives with arXiv scientific papers, using the abstracts as a proxy for gold long-form scientific article summaries. Empirical results from automatic and human evaluations demonstrate that Co-opNet learns to summarize with considerably improved global coherence compared to competitive baselines.
Introduction
While large language models have become incredibly fluent, they still struggle with two major issues in abstractive summarization: factual consistency and narrative flow. Models often "hallucinate" details that weren't in the original text, or they present correct facts in a jarring, illogical order. This lack of discourse structure (e.g., Introduction -> Method -> Results) makes summaries difficult for humans to follow.
The authors argue that standard training objectives, such as maximizing the likelihood of the next word, are insufficient to ensure a summary makes sense as a whole. Furthermore, common datasets like CNN/DailyMail are often too short or too extractive to properly train models on complex narrative flow. This paper addresses these gaps by introducing a framework that explicitly models coherence and factuality.
Methods: The Co-opNet Framework
The authors propose a "generate-then-rerank" architecture called Co-opNet (Cooperative Generator-Discriminator Networks). The system consists of two primary components working in tandem.
The Generator
The generator is a transformer-based language model (specifically GPT-2) fine-tuned to produce a "pool" of potential candidate summaries using various sampling techniques.
The Discriminators (The Critics)
Instead of simply selecting the summary with the highest probability from the generator, Co-opNet employs four specialized discriminator objectives to score the candidates:
- Coverage: Ensures the summary includes necessary discourse roles (e.g., Background, Method, Result).
- Ordering: Evaluates whether these discourse roles appear in a logical, conventional sequence.
- Adjacency: Uses a SciBERT-based model to predict if two sentences logically belong next to each other.
- Factuality: Identifies "salient spans" (key technical terms) and verifies their presence in the source document.
The final summary is chosen by balancing the generator's original probability with the weighted scores from these discriminators.
Experimental Setup
To evaluate narrative flow, the researchers utilized scientific papers, which require stricter logic than news snippets. They used subsets of arXiv (Computer Science and Biology) and the ACL Anthology Network (AAN). Unlike news datasets which often feature 1-2 sentence summaries, these scientific abstracts average 5-6 sentences, providing a robust testbed for evaluating discourse structure.
Key Insights and Results
A central finding of this work is that the generator's "most likely" summary is frequently factually incorrect or poorly structured. As shown in Figure 1, the most coherent and accurate summary often resides further down the list of candidates, necessitating a discriminator to identify it.
Performance Metrics
- Automated Evaluation: Co-opNet consistently outperformed standard baselines, including Pointer-Generator networks, on ROUGE and BERTScore across all scientific domains.
- Human Evaluation: Human judges significantly preferred Co-opNet summaries. Notably, the Adjacency discriminator led to an 18% increase in overall quality compared to the baseline generator.
- Factuality: The factuality discriminator effectively reduced hallucinations, though it occasionally resulted in more extractive summaries to maintain accuracy.
Conclusion
Co-opNet demonstrates that separating generation from coherence evaluation can significantly improve the quality of abstractive summaries. While currently limited by input token constraints (800 tokens) and the quality of the initial candidate pool, the framework provides a path toward more reliable AI-generated text. Future work may involve integrating these discriminator scores directly into the training loop via Reinforcement Learning to further optimize the generation process.
Summary updated