Unsupervised Class-Specific Abstractive Summarization of Customer Reviews

Thi Nhat Anh Nguyen, Mingwei Shen, Karen HovsepianACL 2021

TL;DR

This paper introduces Class-CopyCat, a novel unsupervised method for generating class-specific summaries of customer reviews. It tackles the challenge of producing informative summaries that focus on particular product aspects without relying on human-annotated summaries, making it highly scalable for e-commerce.

What Problem Does This Paper Address?

In the age of e-commerce, customers generate vast amounts of reviews for products. For businesses and consumers alike, sifting through hundreds or thousands of reviews to understand common issues or feedback is a monumental task. While abstractive summarization aims to create concise and fluent summaries, existing unsupervised methods often produce generic content. Furthermore, generating summaries tailored to specific aspects (e.g., "poor quality," "sizing issues") is even more challenging, typically requiring extensive labeled data. This paper addresses the need for unsupervised, class-specific summarization that can distill nuanced feedback from a large volume of reviews into actionable insights for specific product concerns.

Methods

The core of the proposed solution is a Class-CopyCat model, built upon a generative variational autoencoder (VAE) framework. This model is designed to learn class-specific representations from reviews without ground-truth summaries. Key components include:

  • Hierarchical Latent Variable Structure: The model uses latent variables to represent products, classes, and individual reviews, capturing dependencies between them. This allows for class-specific information to be encoded and utilized.
  • Class-Correlation Gate: A novel two-layer filter mechanism is introduced. This gate helps focus on words relevant to a specific class, reducing generic or irrelevant information and enhancing the informativeness of the summaries.
  • Class Probabilities as Signal: Instead of full summaries, the model uses class probabilities (or labels) generated by an independent classifier as the primary training signal. This significantly reduces the annotation burden.
  • Loyalty Term: To combat "hallucinations" (generating information not present in the original reviews), a loyalty term is added to the loss function. This encourages the model to favor words found in the original reviews, improving factual accuracy.

The model's encoder learns representations for product groups, classes, and individual reviews, while the decoder reconstructs reviews. At test time, summaries are generated by fixing latent variables to their mean values and decoding.

Datasets

The experiments were conducted on a subset of the Amazon product reviews dataset (He and McAuley, 2016). Specifically, the authors focused on reviews with 1 to 3-star ratings to capture common issues. They sampled 1000 products for testing and used the rest for training/validation. For evaluation, they collected gold summaries from two external workers for 50 products, with summaries generated for specific issue classes.

Key Insights and Ideas

  • Unsupervised Class-Specific Summarization is Feasible: The paper demonstrates that it's possible to generate class-specific summaries without explicit ground-truth summaries, relying instead on class probabilities. This is a significant step towards scalable opinion summarization.
  • Class-Specific Information is Crucial for Informativeness: Generic summaries are less useful. By explicitly modeling and attending to class-specific information, the model can produce more relevant and insightful summaries.
  • The Class-Correlation Gate is Key: Ablation studies show that the class-correlation gate significantly improves performance, highlighting its importance in filtering and focusing on relevant class-specific content.
  • Loyalty Term Prevents Hallucinations: The loyalty term is crucial for generating factually grounded summaries, preventing the model from inventing information, which is a common issue in abstractive summarization.

Key Results

  • Outperforms Baselines: Class-CopyCat significantly outperforms various abstractive and extractive baselines in terms of ROUGE scores, a standard metric for summary evaluation. For instance, ROUGE-L scores are notably higher than those of CopyCat with class-embedding and collections of CopyCat/MeanSum models.
  • Human Evaluation Wins: Human evaluations show Class-CopyCat achieving the highest average scores across all criteria, including informativeness, conciseness, content support, opinion consensus, fluency, and overall quality. Notably, it achieved a conciseness score of 4.30, far exceeding baselines.
  • Ablation Studies Confirm Component Importance: Removing the class-correlation gate or the loyalty term led to significant drops in performance, underscoring the effectiveness of these novel components.

Limitations and Future Work

  • Dependence on External Classifier: The model's performance is contingent on the quality of the class probabilities provided by an independent classifier. If the classifier is inaccurate, the summaries may suffer.
  • Arbitrary Class Selection: The choice of issue classes is arbitrary and can be expanded. The authors suggest that their model can be applied to any predefined set of topical classes.
  • Future Work: The paper implicitly suggests that exploring more sophisticated class classifiers or integrating the classification task directly into the summarization model could be future directions. Further research into handling a larger number of classes or more complex class hierarchies could also be beneficial.

Summary updated

BibTeX
@inproceedings{nguyen2021unsupervised,
  author = {Thi Nhat Anh Nguyen and Mingwei Shen and Karen Hovsepian},
  title = {{Unsupervised Class-Specific Abstractive Summarization of Customer Reviews}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure