Abstract
In recent years online shopping has gained momentum and became an important venue for customers wishing to save time and simplify their shopping process. A key advantage of shopping online is the ability to read what other customers are saying about products of interest. In this work, we aim to maintain this advantage in situations where extreme brevity is needed, for example, when shopping by voice. We suggest a novel task of extracting a single representative helpful sentence from a set of reviews for a given product. The selected sentence should meet two conditions: first, it should be helpful for a purchase decision and second, the opinion it expresses should be supported by multiple reviewers. This task is closely related to the task of Multi Document Summarization in the product reviews domain but differs in its objective and its level of conciseness. We collect a dataset in English of sentence helpfulness scores via crowd-sourcing and demonstrate its reliability despite the inherent subjectivity involved. Next, we describe a complete model that extracts representative helpful sentences with positive and negative sentiment towards the product and demonstrate that it outperforms several baselines.
Introduction
While customer reviews are a goldmine for shoppers, the sheer volume of data is often overwhelming. Traditional helpfulness metrics typically apply to entire reviews, which are frequently long and contain irrelevant information. In scenarios with limited "real estate"—such as voice interactions with Alexa or quick glances at a smartphone—presenting a full review is impractical.
This paper addresses the gap in current Multi-Document Summarization (MDS) systems, which often struggle with "faithfulness"—ensuring a summary reflects a consensus rather than a one-off outlier. We introduce the task of identifying a Representative Helpful Sentence (RHS): a single sentence that is both helpful for a purchase decision and representative of the collective reviewer opinion.
Methodology
To identify the RHS, we developed an end-to-end pipeline that balances individual sentence utility with collective support. The process consists of four primary stages:
- Helpfulness Estimation: We fine-tuned a BERT model on a newly collected dataset of sentence-level helpfulness scores to predict the utility of standalone sentences.
- Sentiment Analysis: Using Amazon AWS Comprehend, we filter out neutral sentences and categorize the remainder into positive and negative sets, allowing the system to provide both "pro" and "con" perspectives.
- Similarity and Support: To ensure representativeness, we use Sentence-Transformers to identify clusters of semantically similar sentences. A sentence's "support" score is determined by the number of similar neighbors it has in the review pool.
- Sentence Selection: The final RHS is selected by ranking sentences using the formula: $support \times helpful^\alpha$, where $\alpha$ balances the trade-off between helpfulness and consensus.
The Helpful Sentences Dataset
A significant contribution of this work is the Helpful Sentences Dataset, containing 22,000 sentences from 123 products across six diverse categories (Toys, Books, Movies, Music, Camera, and Electronics). Each sentence was rated by 10 to 30 crowd-workers on a 3-level scale. Our analysis found that human annotators can reliably rate sentence helpfulness even without the surrounding review context. Notably, we observed a strong negative correlation (-0.53) between neutral sentiment and helpfulness, suggesting that clear opinions are more valued by consumers.
Key Results
Our BERT-based helpfulness model achieved a Pearson correlation of 0.84 with human scores, significantly outperforming TF-IDF and Ridge regression baselines. In end-to-end ranking tasks (NDCG@1), the RHS model scored 0.87, vastly outperforming "Random" (0.54) and "Longest Sentence" (0.60) baselines.
When compared against state-of-the-art abstractive summarizers like Copycat, our extractive RHS model produced sentences that were nearly as precise as human-selected "Oracle" sentences, despite the extreme constraint of selecting only one sentence.
Conclusion and Future Work
While the model excels at identifying positive RHS sentences, finding supported negative opinions proved more challenging for popular products. Future work may explore abstractive methods to synthesize new sentences that combine elements from multiple similar reviews, potentially further increasing faithfulness. Additionally, live user studies in voice-interface environments would provide valuable insights into real-world customer satisfaction with RHS-based summaries.
Summary updated