Abstract
We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary. Our work addresses current limitations of models for abstractive summarization that often hallucinate information or generate summaries with coherence issues. To generate abstractive summaries with factual consistency and narrative flow, we propose Cooperative Generator -- Discriminator Networks (Co-opNet), a novel transformer-based framework where a generator works with a discriminator architecture to compose coherent long-form summaries. We explore four different discriminator objectives which each capture a different aspect of coherence, including whether salient spans of generated abstracts are hallucinated or appear in the input context, and the likelihood of sentence adjacency in generated abstracts. Empirical results from automatic and human evaluations demonstrate that Co-opNet learns to summarize with considerably improved global coherence compared to competitive baselines.
Introduction
Abstractive summarization—the task of generating a summary in a model's own words—remains a significant challenge in Natural Language Processing (NLP). While current state-of-the-art models are proficient at generating fluent English sentence-by-sentence, they frequently suffer from two critical flaws: hallucinations and discourse failure. Hallucinations occur when models confidently state facts not present in the source text, while discourse failure refers to a lack of logical "flow," where summaries may jump disjointedly between sections like methods and results.
Standard language models are optimized for next-word prediction, which does not necessarily align with the high-level narrative arc required for a coherent paragraph. This paper introduces Co-opNet to bridge this gap by explicitly modeling discourse and factuality.
The Co-opNet Framework
The proposed Cooperative Generator–Discriminator Networks (Co-opNet) framework operates through a writer-editor relationship.
- The Generator: A transformer-based model (GPT-2) generates a pool of potential candidate summaries for a given scientific introduction.
- The Discriminator: A separate SciBERT-based model evaluates these candidates based on four coherence objectives:
- Coverage: Ensuring all necessary sections (Background, Methods, Results) are present.
- Ordering: Verifying the logical sequence of these sections.
- Adjacency: Checking if consecutive sentences belong together.
- Factuality: Confirming that "salient spans" (technical terms and claims) are grounded in the source document.
The final summary is selected by reranking the generator's candidates using the discriminator's scores.
Experimental Setup and Datasets
To test the model's ability to maintain narrative flow, the researchers utilized scientific paper datasets where discourse structure is rigorous:
- arXiv: Computer Science (CS) and Biology (Bio) subsets, totaling over 50,000 samples.
- AAN (ACL Anthology Network): Approximately 12,000 papers from the computational linguistics domain.
These datasets use the original paper abstracts as a proxy for "gold" long-form summaries, providing a high bar for coherence and technical accuracy.
Results and Key Insights
A primary finding of this research is that Probability ≠ Quality. In standard models, the summary with the highest log-likelihood is often factually incorrect or poorly structured. Co-opNet's discriminator often identifies better summaries that are buried further down the list of candidates.
Empirical results highlight the effectiveness of this approach:
- Automatic Metrics: On the AAN dataset, Co-opNet improved ROUGE-L scores by over 12% compared to Pointer-Generator (PGen) baselines.
- Human Evaluation: Human judges preferred Co-opNet summaries for coherence significantly more often than the baseline (34.01% vs 23.81% for the Adjacency model).
- Factuality: The "Salient Span" approach allowed the model to focus on critical technical phrases, effectively reducing the hallucination rate compared to standard transformers.
Conclusion
Co-opNet demonstrates that discourse understanding is a powerful proxy for truth. By modeling how a story is told, the framework generates summaries that are both more coherent and more factually grounded. Future work may involve expanding the discriminator pool to include domain-specific constraints or addressing the limitations of fixed input context windows in transformer architectures.
Summary updated