Extractive Summarization via ChatGPT for Faithful Summary Generation

Haopeng Zhang, Xiao Liu, Jiawei Zhang

Introduction

Large Language Models (LLMs), particularly those in the GPT family, have redefined the state-of-the-art in natural language generation. In the context of text summarization, these models typically operate in an abstractive mode, paraphrasing and synthesizing information to create fluent summaries. However, this flexibility comes with a significant drawback: the tendency to hallucinate. Abstractive models often generate factually incorrect statements or misinterpret the source material, which limits their utility in high-stakes domains like medicine or legal analysis.

This research investigates an alternative approach: leveraging ChatGPT for extractive summarization. By selecting sentences directly from the source text rather than generating new ones, the model can theoretically maintain higher factual consistency. This paper explores various prompting strategies to optimize ChatGPT's extractive capabilities and proposes a hybrid "Extract-then-Generate" pipeline to bridge the gap between faithfulness and fluency.

Methodology

The study evaluates the performance of gpt-3.5-turbo using four distinct prompting strategies designed to elicit extractive behavior:

  1. Zero-shot Extractive Summarization: The model is prompted to select a specific number of sentences ($m$) from the source document without any prior examples.
  2. In-Context Learning (ICL): The model is provided with a small set of document-summary pairs (few-shot) to demonstrate the desired extractive style before processing the target document.
  3. Chain-of-Thought (CoT): This strategy requires the model to generate a rationale or "reason" for selecting specific sentences before providing the final extraction. This encourages the model to engage in explicit reasoning about document saliency.
  4. Extract-then-Generate Pipeline: This is a two-stage framework. In the first stage, ChatGPT performs an extractive summary to identify the most critical sentences. In the second stage, the model uses only these extracted sentences as the source to generate a polished, abstractive summary.

Experimental Setup

To ensure a robust evaluation, the researchers utilized four diverse datasets representing different writing styles and domains:

  • CNN/DailyMail: A standard news dataset characterized by bullet-point highlights.
  • XSum: A dataset requiring highly abstractive, single-sentence news summaries.
  • PubMed: A long-form scientific dataset using paper introductions as sources and abstracts as targets.
  • Reddit (TIFU): An informal, conversational dataset from social media posts.

For each dataset, a sample of 1,000 examples was used for evaluation due to the computational costs associated with LLM APIs.

Results and Discussion

The ROUGE Paradox and G-EVAL

A primary finding of the study is the discrepancy between traditional n-gram overlap metrics and LLM-based evaluations. ChatGPT consistently achieved lower ROUGE scores compared to smaller, fine-tuned extractive models (e.g., 39.25 vs 44.41 on CNN/DM). However, when evaluated using G-EVAL—a metric that uses LLMs to mimic human judgment—ChatGPT outperformed the fine-tuned baselines. This suggests that while ChatGPT may not select the exact same words as human annotators (the "gold standard"), the sentences it chooses are perceived as high-quality and salient by human-like logic.

Enhancing Faithfulness

The most significant contribution of the "Extract-then-Generate" pipeline is the drastic reduction in hallucinations. Purely abstractive summaries from ChatGPT showed surprisingly low faithfulness, with FactCC scores (a metric for factual consistency) falling below 10% on certain datasets. By constraining the generation process to a pre-extracted set of sentences, the FactCC scores improved by over 500% in some instances, rising from 6.35% to 51.65% on the CNN/DM dataset.

Positional and Reasoning Biases

The study identified a persistent "lead bias" in ChatGPT’s extractions. The model shows a strong preference for sentences at the beginning of a document, a tendency that exceeds even human-written summaries. Furthermore, while Chain-of-Thought reasoning improved performance on difficult datasets like XSum by helping the model understand saliency patterns, it did not entirely eliminate the model's inherent biases toward document structure.

Conclusion

This research demonstrates that while ChatGPT is primarily known for its abstractive prowess, it can be effectively harnessed for extractive summarization to ensure factual integrity. The "Extract-then-Generate" pipeline offers a promising solution: the factual grounding of extractive methods combined with the linguistic polish of abstractive generation. Future work should expand this analysis to more advanced models like GPT-4 and explore methods to mitigate the lead bias observed in longer documents.

Summary updated

BibTeX
@inproceedings{zhang2023extractive,
  author = {Haopeng Zhang and Xiao Liu and Jiawei Zhang},
  title = {{Extractive Summarization via ChatGPT for Faithful Summary Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.214},
}
Figure