Zero-Shot Opinion Summarization with GPT-3

Adithya Bhaskar, Alexander R. Fabbri, Greg DurrettarXiv 2022 · OpenReview

Introduction

Opinion summarization represents a unique challenge in the field of natural language processing. Unlike news summarization, where key information is typically concentrated at the beginning of a text, user reviews (e.g., for hotels or consumer products) are characterized by a high volume of text and frequently conflicting viewpoints. A single hotel might receive praise for its staff while simultaneously being criticized for its cleanliness. Capturing this "he said, she said" dynamic while managing the limited context windows of Large Language Models (LLMs) like GPT-3.5 is the central problem addressed by Bhaskar et al. (2023).

Methodology: Pipelined Summarization

To handle datasets that exceed the input limits of GPT-3.5, the researchers evaluated several pipelined strategies designed to distill large quantities of reviews into coherent summaries.

One of the primary methods explored is the TCG (Topic-Clustering-Generation) pipeline. As illustrated in Figure 1, this approach involves three distinct stages: first, classifying individual sentences by topic (e.g., service, food, rooms); second, summarizing these topical chunks; and finally, generating a cohesive summary from those intermediate summaries.

The TCG pipeline: Topic classification, chunk summarization, and final generation.

Other methods tested include:

  • Recursive Chunking: A hierarchical approach where reviews are summarized in small groups, and those summaries are themselves summarized until a final output is reached.
  • QG (Pre-extraction + GPT): This hybrid approach uses a specialized extractive model, QFSumm, to identify the 35 most relevant sentences across all reviews. These sentences are then fed into GPT-3.5 for a final abstractive pass. Surprisingly, this "pre-filtering" often led to more faithful summaries than recursive methods, which tended to lose specific details through successive layers of abstraction.

Datasets and Evaluation

The study utilized two benchmark datasets: SPACE, which focuses on aspect-oriented hotel reviews (e.g., summarizing only the "location" aspect), and FewSum, which contains generic reviews from Amazon and Yelp.

A critical contribution of this work is the critique of standard evaluation metrics. The authors demonstrate that ROUGE scores, which rely on word overlap, have almost zero correlation with human judgments when evaluating LLM-generated summaries. Instead, they propose a framework using Natural Language Inference (NLI). By breaking summaries into atomic propositions (e.g., "The room was clean") and checking if the original reviews support these claims, they can measure "Faithfulness" (how accurately the summary reflects the consensus) and "Factuality" (whether the summary makes up non-existent facts).

Key Results and the "Fluency Trap"

Human evaluations revealed a strong preference for GPT-3.5 pipelines, particularly TCG, over previous state-of-the-art models like AceSum. As shown in Figure 2, the TCG output is highly fluent and covers multiple nuances of the "Rooms" aspect, such as space and cleanliness, while also noting specific complaints like AC noise.

Comparison of summaries for the 'Rooms' aspect, showing TCG's fluency and detail.

However, the researchers identified a significant risk termed the "Fluency Trap." Because GPT-3.5 is exceptionally good at writing, it can easily "hallucinate consensus." It might use authoritative phrasing like "Most reviewers enjoyed..." to describe a sentiment that appeared in only a single review. This leads to high factuality (the claim exists in the source) but low faithfulness (the claim is misrepresented as a majority opinion).

Conclusion

While GPT-3.5 significantly advances the state of the art in terms of summary relevance and fluency, the issue of "consensus hallucination" remains a hurdle for zero-shot opinion summarization. The study suggests that future work should focus on models that can explicitly track opinion prevalence—essentially "counting" the weight of different viewpoints—to ensure that the final summary accurately reflects the true distribution of user sentiment rather than just the most interesting or fluent claims.

Summary updated

BibTeX
@misc{bhaskar2022zero,
  author = {Adithya Bhaskar and Alexander R. Fabbri and Greg Durrett},
  title = {{Zero-Shot Opinion Summarization with GPT-3}},
  year = {2022},
  url = {https://openreview.net/forum?id=UHpUp9r0tCE},
}
Figure