Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, Wei ChengarXiv 2023 · DOI · arXiv

Introduction

While large language models (LLMs) like ChatGPT have demonstrated remarkable proficiency in generic text summarization, their efficacy in specialized tasks—specifically query-based and aspect-based summarization—remains a critical area of investigation. Generic summarization aims for a broad overview, but real-world applications often require models to synthesize information through a specific lens, such as answering a user query or focusing on a particular emotional aspect. This study evaluates whether ChatGPT's general-purpose training can compete with or surpass traditional models fine-tuned on task-specific datasets.

Datasets and Task Diversity

To rigorously test ChatGPT's versatility, the researchers utilized four diverse benchmark datasets covering different domains and summarization types. These range from short Reddit posts to long-form meeting transcripts and stories.

Characteristics of the four benchmark datasets used in the study, highlighting the diversity in domains and input lengths.

The datasets include:

  1. QMSum: Meeting transcripts requiring query-based summaries.
  2. SQuALITY: Long-form English stories requiring query-based summaries.
  3. CovidET: Reddit posts about COVID-19, requiring aspect-based summaries focused on emotional triggers.
  4. NEWTS: News articles requiring aspect-based summaries on specific topics.

Methodology

The evaluation focused on ChatGPT (early 2023 version) using several prompting strategies:

  • Zero-shot Prompting: Directly asking for a summary based on a query or aspect.
  • One-shot Prompting: Used for datasets like CovidET where strict length constraints were difficult for the model to meet without an example.
  • Extract-then-Summarize: A two-step pipeline for long documents (e.g., QMSum) that exceed the model's token limit, involving an initial retrieval of relevant spans followed by summarization.

Performance was measured using standard ROUGE scores and lexical metrics (Coverage, Density, Compression) to analyze stylistic differences.

Key Results

The study found that ChatGPT is a formidable competitor to fine-tuned models, often matching or exceeding their performance despite not being explicitly trained on these specific tasks.

Comparison of ROUGE scores between fine-tuned models and ChatGPT across different datasets.

Notably, on the NEWTS dataset, ChatGPT outperformed fine-tuned models in ROUGE-1 (32.54 vs. 31.78). In the QMSum (Golden) setting—where the model is provided with the specific relevant parts of a meeting—ChatGPT also surpassed fine-tuned models in both ROUGE-1 and ROUGE-2, demonstrating superior synthesis capabilities when noise is minimized.

Discussion and Insights

The ROUGE Gap and Stylistic Differences

A significant finding is that ROUGE scores may underestimate ChatGPT's quality. ChatGPT tends to produce more formal, abstractive, and "extended" summaries. In contrast, fine-tuned models often learn to mimic the specific, sometimes idiosyncratic, style of human references in a dataset to maximize overlap scores. Qualitative analysis suggested that ChatGPT's summaries are often more fluent and professional than the provided references.

Abstractive Nature and Length Sensitivity

ChatGPT is highly abstractive, preferring to rephrase and synthesize rather than extract sentences verbatim. It also exhibits a "natural" output length; when forced into extremely brief summaries (like the one-sentence requirements in CovidET), its performance improves significantly with one-shot prompting to calibrate its output.

Handling Long Contexts

For documents exceeding 3,000 words, such as those in SQuALITY or QMSum, the "Extract-then-Summarize" approach or simple truncation proved effective. However, the results suggest that retrieval-augmented generation is the most promising path for handling truly long-form content.

Conclusion

ChatGPT demonstrates a "superpower" for synthesizing information across diverse domains. While it may be penalized by n-gram-based metrics like ROUGE due to its abstractive and formal style, its zero-shot and one-shot performance indicates that general-purpose LLMs are rapidly closing the gap with specialized, fine-tuned experts in complex summarization tasks. Future work should focus on more nuanced human evaluations and multi-turn prompting strategies to further unlock these models' potential.

Summary updated

Abstract

Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various methods have been proposed for text summarization, including extractive and abstractive summarization. The emergence of large language models (LLMs) like GPT3 and ChatGPT has recently created significant interest in using these models for text summarization tasks. Recent studies \cite{goyal2022news, zhang2023benchmarking} have shown that LLMs-generated news summaries are already on par with humans. However, the performance of LLMs for more practical applications like aspect or query-based summaries is underexplored. To fill this gap, we conducted an evaluation of ChatGPT's performance on four widely used benchmark datasets, encompassing diverse summaries from Reddit posts, news articles, dialogue meetings, and stories. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores. Moreover, we highlight some unique differences between ChatGPT-generated summaries and human references, providing valuable insights into the superpower of ChatGPT for diverse text summarization tasks. Our findings call for new directions in this area, and we plan to conduct further research to systematically examine the characteristics of ChatGPT-generated summaries through extensive human evaluation.
BibTeX
@misc{yang2023exploring,
  author = {Xianjun Yang and Yan Li and Xinlu Zhang and Haifeng Chen and Wei Cheng},
  title = {{Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization}},
  year = {2023},
  eprint = {2302.08081},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.08081},
}
Figure