Introduction
While large language models (LLMs) like ChatGPT have demonstrated remarkable proficiency in generic text summarization, their efficacy in specialized tasks—specifically query-based and aspect-based summarization—remains a critical area of investigation. Generic summarization aims for a broad overview, but real-world applications often require models to synthesize information through a specific lens, such as answering a user query or focusing on a particular emotional aspect. This study evaluates whether ChatGPT's general-purpose training can compete with or surpass traditional models fine-tuned on task-specific datasets.
Datasets and Task Diversity
To rigorously test ChatGPT's versatility, the researchers utilized four diverse benchmark datasets covering different domains and summarization types. These range from short Reddit posts to long-form meeting transcripts and stories.
The datasets include:
- QMSum: Meeting transcripts requiring query-based summaries.
- SQuALITY: Long-form English stories requiring query-based summaries.
- CovidET: Reddit posts about COVID-19, requiring aspect-based summaries focused on emotional triggers.
- NEWTS: News articles requiring aspect-based summaries on specific topics.
Methodology
The evaluation focused on ChatGPT (early 2023 version) using several prompting strategies:
- Zero-shot Prompting: Directly asking for a summary based on a query or aspect.
- One-shot Prompting: Used for datasets like CovidET where strict length constraints were difficult for the model to meet without an example.
- Extract-then-Summarize: A two-step pipeline for long documents (e.g., QMSum) that exceed the model's token limit, involving an initial retrieval of relevant spans followed by summarization.
Performance was measured using standard ROUGE scores and lexical metrics (Coverage, Density, Compression) to analyze stylistic differences.
Key Results
The study found that ChatGPT is a formidable competitor to fine-tuned models, often matching or exceeding their performance despite not being explicitly trained on these specific tasks.
Notably, on the NEWTS dataset, ChatGPT outperformed fine-tuned models in ROUGE-1 (32.54 vs. 31.78). In the QMSum (Golden) setting—where the model is provided with the specific relevant parts of a meeting—ChatGPT also surpassed fine-tuned models in both ROUGE-1 and ROUGE-2, demonstrating superior synthesis capabilities when noise is minimized.
Discussion and Insights
The ROUGE Gap and Stylistic Differences
A significant finding is that ROUGE scores may underestimate ChatGPT's quality. ChatGPT tends to produce more formal, abstractive, and "extended" summaries. In contrast, fine-tuned models often learn to mimic the specific, sometimes idiosyncratic, style of human references in a dataset to maximize overlap scores. Qualitative analysis suggested that ChatGPT's summaries are often more fluent and professional than the provided references.
Abstractive Nature and Length Sensitivity
ChatGPT is highly abstractive, preferring to rephrase and synthesize rather than extract sentences verbatim. It also exhibits a "natural" output length; when forced into extremely brief summaries (like the one-sentence requirements in CovidET), its performance improves significantly with one-shot prompting to calibrate its output.
Handling Long Contexts
For documents exceeding 3,000 words, such as those in SQuALITY or QMSum, the "Extract-then-Summarize" approach or simple truncation proved effective. However, the results suggest that retrieval-augmented generation is the most promising path for handling truly long-form content.
Conclusion
ChatGPT demonstrates a "superpower" for synthesizing information across diverse domains. While it may be penalized by n-gram-based metrics like ROUGE due to its abstractive and formal style, its zero-shot and one-shot performance indicates that general-purpose LLMs are rapidly closing the gap with specialized, fine-tuned experts in complex summarization tasks. Future work should focus on more nuanced human evaluations and multi-turn prompting strategies to further unlock these models' potential.
Summary updated