Abstract
Opinion summarization involves distilling hundreds of potentially conflicting user reviews into a concise consensus. While Large Language Models (LLMs) like GPT-3.5 offer high fluency, they face significant challenges regarding context limits, faithfulness, and the "echo effect." This paper introduces specialized pipelines to handle long-context review data and proposes new entailment-based metrics that outperform standard ROUGE scores in detecting factual errors and over-generalizations.
Introduction
Summarizing opinions, such as hotel or product reviews, differs fundamentally from news summarization. Unlike news, where key information is often front-loaded, review summarization requires balancing contradictory viewpoints (e.g., "firm bed" vs. "soft bed") and synthesizing a representative consensus.
The authors identify three primary hurdles for LLMs in this domain:
- Context Limits: Large review collections often exceed the model's maximum input length.
- The Echo Effect: Models may extract and "echo" input sentences rather than synthesizing them.
- Evaluation Gap: Standard metrics like ROUGE and BERTScore fail to catch "faithfulness" errors, such as misrepresenting a single complaint as a majority view.
Methods and Architectures
To address these challenges, the researchers explored several pipeline architectures designed to bridge the gap between raw reviews and a final summary.
Pipeline Architectures
- TCG (Topic-Chunk-Generate): A three-step process where sentences are first clustered by topic (e.g., "cleanliness," "service"), then recursively chunked and summarized until the text fits the model's window, followed by a final generation step.
- Extractive Filtering (QG): This approach uses a specialized extractive model (QFSumm) to select the top 35 most salient sentences before passing them to GPT-3.5 for abstractive polishing.
- Review Stratification (RG): Reviews are grouped by star ratings to ensure the summary reflects both positive and negative extremes.
Evaluation Framework
To measure factuality, the authors implemented an Atomic Evaluation method. GPT-3.5 is used to break complex summary sentences into simple, atomic propositions. These propositions are then verified against source reviews using a Natural Language Inference (NLI) model to determine if they are supported by the original text.
Datasets
The study utilized two primary benchmarks: the SPACE dataset (100 hotel reviews per entity) and FewSum (Amazon and Yelp reviews). While SPACE focuses on specific aspects like service or location, FewSum is generic and typically features shorter input lengths.
Key Results and Insights
The study revealed that GPT-3.5 often falls into a "Generalization Trap," where it turns a single reviewer's niche complaint into a general statement. This subtle hallucination is difficult for standard metrics to detect.
Quantitative and Qualitative Findings
- Metric Correlation: The proposed entailment-based metrics showed a Spearman correlation of 0.36 for factuality, whereas ROUGE-L showed a negative correlation (-0.03) with human faithfulness ratings. This confirms that ROUGE is often "truth-blind" in the context of opinion summarization.
- Human Preference: Despite lower ROUGE scores compared to some specialized extractive models, the TCG and QG pipelines were highly rated by humans for fluency and relevance.
- Recursive Dilution: While recursive chunking (TCG) handles long inputs, each layer tends to make the output more generic. Single-step extractive filtering (QG) often preserved better factual grounding.
Conclusion
While GPT-3.5 rarely invents facts entirely, it struggles with "representativeness"—correctly identifying majority views versus outliers. Future work should focus on fine-grained control over summary genericity and developing models capable of explicitly citing source reviews to support their claims.
Summary updated