TL;DR
This paper introduces a novel system for generating concise and informative product review summaries by combining aspect-based sentiment analysis (ABSA) with large language models (LLMs). The system effectively addresses the challenge of information overload in e-commerce reviews, leading to improved user engagement metrics in a large-scale A/B test.
What Problem Does This Paper Address?
In the age of e-commerce, customer reviews are a goldmine of information, offering invaluable insights into product quality and user experience. However, the sheer volume and often repetitive nature of these reviews can make it incredibly difficult for potential buyers to quickly find the most relevant information. This "information overload" can lead to missed opportunities for customers to discover products that truly meet their needs. While Large Language Models (LLMs) offer a promising solution for automatically summarizing these reviews, they are prone to issues like hallucination, omission of critical facts, and misrepresentation, especially when dealing with vast and noisy datasets. Existing summarization methods often struggle with scalability, the cost of creating labeled data, and adapting to evolving review content. This paper tackles the challenge of creating accurate, interpretable, and scalable review summaries that are grounded in actual customer feedback.
Methods
The core of the Wayfair system is an end-to-end pipeline that leverages LLMs for aspect-guided review summarization. The process involves several key stages:
- Aspect Extraction: Individual customer reviews are processed by a pre-trained LLM to identify up to five relevant product aspects and their associated sentiment (positive, mixed, or negative). This is achieved through structured prompts that guide the LLM to output a JSON object for each review.
- Aspect Consolidation: To handle the natural language variability in aspect terms (e.g., "value for money" vs. "price"), a second LLM-based stage consolidates fine-grained aspects into broader, canonical forms. This step ensures a consistent vocabulary and uses a frequency threshold to preserve well-established aspects while merging less frequent ones.
- Aspect-Based Review Selection: The system identifies the top five most frequently mentioned aspects for each product. Representative reviews for each aspect-sentiment pair are then sampled to create a focused input for the summarization model. The number of input reviews is capped to manage context length.
- Aspect-Guided Summarization: Finally, the LLM generates a concise, product-level summary. This summary is designed to cover the most frequent product aspects and accurately reflect the underlying customer feedback, aiming for a natural language text of approximately 300-500 characters.
The entire pipeline is designed to be modular and model-agnostic, allowing for the use of different LLMs and customization of individual components. The authors specifically use Gemini 1.5 Flash for their LLM-based tasks.
Datasets
The paper introduces a new, open-sourced dataset to support research in aspect-guided review summarization. This dataset is a subset of the anonymized production data used in their A/B test and includes:
- 92,000 products from the 1000 most frequent product classes.
- A total of 11.8 million reviews, with each product having between 100 and 300 reviews.
- An average review length of 124 characters and an average of 129 reviews per product.
- Crucially, this dataset pairs customer reviews with high-quality, production-level summaries and includes extracted aspect-sentiment pairs (after consolidation), differentiating it from datasets that only provide raw reviews.
Key Insights and Ideas
- LLMs as a powerful tool for structured summarization: The paper demonstrates that LLMs, when guided by aspect-sentiment analysis, can go beyond generic summarization to produce highly relevant and interpretable summaries grounded in specific product features.
- Aspect consolidation is crucial for consistency: The variability of natural language in customer reviews necessitates a consolidation step to create a unified understanding of product aspects, which is vital for accurate summarization.
- Real-world impact matters: The large-scale A/B test provides strong evidence that aspect-guided summaries not only improve the linguistic quality of summaries but also have a tangible positive impact on user engagement metrics.
- Scalability through modularity and caching: The system's design, incorporating modular components and cached aspect mappings, allows for efficient real-time deployment and adaptation to evolving data.
Key Results
- Offline Evaluation: 84% of the generated summaries had no errors. Minor issues were found in 11% of summaries, and major issues in 5%. The evaluation revealed strong factual consistency and alignment with extracted aspects.
- Online A/B Test: The introduction of aspect-guided summaries led to a statistically significant 0.3% increase in Visit-Level Add to Cart Rate (ATCR). Secondary metrics also improved, with a 0.5% increase in Conversion Rate (CVR) and a 0.13% decrease in customer-level bounce rate. No negative impacts on gross revenue or page speed were observed.
Limitations and Future Work
- LLM Dependency: The performance of the pipeline is inherently tied to the quality of the underlying LLM, impacting accuracy, fluency, and alignment with customer sentiment.
- Rare Product Classes: Performance may degrade for product classes with limited review coverage or highly domain-specific terminology.
- Future Work: The authors suggest exploring improved robustness across a broader range of domains, languages, and product categories. They also aim for better alignment between extracted aspects and user intent. Further research could also focus on more sophisticated methods for handling rare aspects and domain-specific terminology.
Summary updated