SimsterQ: A Similarity based Clustering Approach to Opinion Question Answering

Aishwarya Ashok, Ganapathy Natarajan, Ramez Elmasri, Laurel Smith-StvanACL 2020

TL;DR

This paper introduces SimsterQ, an unsupervised clustering system for answering opinion-based questions using customer reviews. It demonstrates that a simple, unsupervised approach can achieve performance comparable to or even exceeding more complex deep learning methods, offering a computationally efficient alternative for opinion QA.

What Problem Does This Paper Address?

The explosion of online shopping has led to a massive volume of customer reviews. When users want to understand specific aspects of a product, sifting through thousands of reviews to find relevant opinions is incredibly time-consuming. Traditional Question Answering (QA) systems are designed for factual queries, not for extracting subjective opinions. This paper tackles the challenge of opinion QA by developing a system that can effectively mine customer reviews to answer binary (yes/no) questions, helping users make informed purchasing decisions without the manual effort of reading through extensive review data.

Methods

The core of SimsterQ is an unsupervised clustering approach. The system takes a question and a set of product reviews as input and follows these steps:

  1. Relevant Review Selection: It first identifies reviews associated with the specific product in question.
  2. Sentence-Level Similarity: Reviews are preprocessed (removing punctuation and HTML tags), and then each sentence is converted into a vector representation using word2vec. The cosine similarity between each sentence vector and the question vector is calculated.
  3. Filtering: Sentences with a cosine similarity below a threshold (0.5) are filtered out to remove irrelevant candidates.
  4. Clustering: The remaining sentences are ordered by similarity. The system then forms clusters by grouping sentences with high cosine similarity (threshold 0.9) to each other. This process is iterative, and the algorithm naturally determines the number of clusters.
  5. Top-k Sentence Selection: The top 10 clusters (those with the highest similarity scores) are selected. From each of these clusters, either the first sentence or the sentence with the median length is chosen as a potential answer.

The paper evaluates five variants of this system:

  • sim: Uses cosine similarity with all words.
  • sim_ns: Uses cosine similarity without stopwords.
  • sim_med: Uses the median-length sentence from a cluster.
  • sim_med_ns: Uses the median-length sentence without stopwords.
  • 3g: A 3-gram method that splits questions into 3-grams to capture meaning more effectively.

Datasets

The primary dataset used is AmazonQA (Gupta et al., 2019), which contains both binary and open-ended questions along with review snippets. This dataset was built upon previous work by Wan and McAuley (2016). The authors also utilized the Amazon Reviews Dataset, which contains over 142 million reviews across 24 product categories, to source the reviews for their system. For evaluation purposes, they randomly selected 1000 questions (200 from each of five categories: Automotive, Baby, Beauty, Pet Supplies, and Tools & Home Improvement).

Key Insights and Ideas

  • Unsupervised Simplicity Outperforms Complexity: The paper's most striking finding is that a straightforward, unsupervised clustering method (SimsterQ) can achieve performance comparable to, or even better than, more sophisticated deep learning models. This suggests that for opinion QA, the complexity of the model might not always translate to superior results.
  • Clustering as a Natural Answer Selector: The algorithm's ability to self-select the appropriate number of clusters is a key advantage. Instead of pre-defining a fixed number of clusters (like k-means), SimsterQ dynamically groups similar sentences, adapting to the inherent structure of the review data for each question.
  • Cosine Similarity as a Robust Metric: The paper emphasizes cosine similarity, derived from word vectors, as a more appropriate metric for opinion QA than traditional metrics like ROUGE, especially when dealing with subjective content.
  • Gold Standard Nuances: The authors acknowledge the difficulty of establishing a perfect gold standard for opinion QA. They use Okapi BM25 as a benchmark for sentence-level evaluation and review snippets from the AmazonQA dataset for answer-level evaluation, recognizing that these might not perfectly align with human judgment of "best" answers.

Key Results

  • Sentence Level: The sim variant of SimsterQ consistently performed best across most metrics, including ROUGE-1, ROUGE-L, and Similarity Accuracy. It achieved a Similarity Accuracy of 91.5% (sentences with >0.7 cosine similarity to gold standard) and a Correct Answer rate of 83.6% (questions with at least one answer >0.7 similarity). The system outperformed the R-Net baseline (ROUGE-L F-score of 40.22) used by Gupta et al. (2019).
  • Answer Level: At the answer level, SimsterQ also showed strong performance. The sim variant achieved a ROUGE-L F-score of 29.66, and a Similarity Accuracy of 95.94%. This indicates that the top candidate sentences returned by SimsterQ are highly similar to the review snippets provided as the gold standard.
  • Cluster Quality: The algorithm naturally selected between 2 and 6 clusters for most questions, and cluster quality metrics (Silhouette score and Calinski-Harabasz score) indicated good performance. Approximately 80% of questions had an appropriate number of clusters identified by the algorithm.

Limitations and Future Work

  • Binary Questions Focus: The current system is primarily evaluated on binary (yes/no) questions. The authors acknowledge that extending SimsterQ to handle open-ended questions presents a significant challenge, particularly in evaluation.
  • Gold Standard Alignment: The paper notes that the "gold standard" answers (review snippets) may not always perfectly align with the user's perspective or the ideal answer. SimsterQ aims to provide a new gold standard for binary questions.
  • Future Directions: The authors plan to extend SimsterQ to handle open-ended questions. They also intend to evaluate the system on other datasets to assess its generalizability and explore its use in generating new gold standards for binary questions.

Summary updated

BibTeX
@inproceedings{ashok2020simsterq,
  author = {Aishwarya Ashok and Ganapathy Natarajan and Ramez Elmasri and Laurel Smith-Stvan},
  title = {{SimsterQ: A Similarity based Clustering Approach to Opinion Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure