Deciphering word-of-mouth in social media: Text-based metrics of consumer reviews

Zhu Zhang, Xin Li, Yubo ChenACM Trans. Manag. Inf. Syst 2012 · DOI

TL;DR

This paper introduces novel "sentiment divergence" metrics to better understand how consumer reviews on social media influence product sales. Unlike traditional sentiment analysis that focuses on overall positivity or negativity, these new metrics capture the spread of opinions, revealing that a wider range of sentiments (even including negative ones) can actually boost sales by providing more nuanced product-matching information.

What Problem Does This Paper Address?

In today's digital age, consumers heavily rely on online reviews and social media for purchasing decisions. While businesses recognize the importance of "word-of-mouth" (WOM), understanding what about these reviews drives sales has been challenging. Existing research often focuses on simple metrics like the number of reviews or average star ratings. However, these can be superficial. For instance, a product with many highly positive reviews might still not sell as well as one with a mix of positive and negative feedback, because the latter provides more detailed information about how the product fits different user needs. This paper aims to bridge this gap by developing and validating text-based metrics that go beyond simple sentiment valence to capture the diversity of opinions expressed in reviews, and to demonstrate their impact on sales.

Methods

The core innovation of this paper lies in its "sentiment divergence" metrics. These metrics are derived from analyzing the textual content of consumer reviews. The process involves:

  1. Word Sentiment Quantification: Using SentiWordNet, each word in a review is assigned a sentiment score (positive, negative, or neutral) based on its positivity and negativity scores.
  2. Microstate Sequence Generation: Each review is converted into a sequence of these sentiment scores, representing the sentiment flow within the review.
  3. Probability Distribution Mapping: These microstate sequences are then mapped to probability distributions.
  4. Divergence Measurement: Two key divergence metrics are proposed:
  5. Sentiment KL Divergence (SentiDvg KL): This measures the average pairwise symmetrized Kullback-Leibler divergence between the sentiment distributions of all reviews for a given product. It quantifies how much the opinions differ from each other.
  6. Sentiment JS Divergence (SentiDvg JS): This uses the Generalized Jensen-Shannon Divergence to compare each review's sentiment distribution to an "average" sentiment distribution for the product. It quantifies how much individual opinions deviate from the center.

These metrics are then incorporated into econometric models (specifically, a Difference-in-Difference approach) to assess their impact on product sales, controlling for other factors like price, volume, and average ratings.

Datasets

The study utilizes a longitudinal dataset collected from Amazon.com and BN.com (Barnes & Noble) for books within the "Multicultural" subcategory of the "Romance" genre. Data was collected over three time periods: mid-November 2009, mid-January 2010, and mid-July 2010. For each book, the dataset includes product information, sales rank, price, number of reviews, average rating, standard deviation of ratings, and the textual content of all product reviews.

Key Insights and Ideas

  • Divergence Over Valence: The paper strongly suggests that the spread or divergence of opinions in reviews is more critical for sales than just the overall positive or negative sentiment (valence).
  • Negative Reviews Can Be Good: Negative reviews, when part of a diverse set of opinions, can actually be beneficial by providing crucial product-matching information, reducing uncertainty for potential buyers.
  • Beyond Numerical Ratings: Sentiment divergence metrics capture nuances that simple star ratings or average scores miss, offering a richer understanding of consumer sentiment.
  • Informative vs. Persuasive: While average ratings might persuade, divergent reviews inform, helping consumers make better decisions and ultimately leading to more sales.
  • Time-Dependent Effects: The impact of rating variance on sales can shift over time (from positive to negative), whereas sentiment divergence appears to have a more consistently positive effect.

Key Results

  • Significant Sales Impact: The proposed sentiment divergence metrics (SentiDvg KL and SentiDvg JS) were found to have a significant positive impact on product sales, even after controlling for traditional metrics like average ratings and review volume.
  • Superiority to Existing Metrics: Models incorporating sentiment divergence metrics showed a significantly better fit and more significant coefficients compared to models using only nontextual measures or simpler textual sentiment measures (like subjectivity or average valence).
  • Hypothesis Support: The results supported the hypotheses that higher sentiment divergence leads to larger sales impacts (H1) and that the impact of sentiment divergence is more stable over time compared to rating variance (H2b).
  • Rating Variance Dynamics: The study confirmed that the impact of rating variance on sales can shift from positive to negative over time (H2a).

Limitations and Future Work

  • Lexical Approach: The current sentiment divergence metrics rely on a lexicon-based approach (SentiWordNet) and shallow lexical information. The authors acknowledge that more sophisticated natural language processing techniques could potentially capture more nuanced information and lead to stronger sales effects or better prediction.
  • Self-Selection Bias: The study acknowledges the potential for self-selection bias in online reviews (early adopters' opinions might differ from the broader population). Future work could explore computing variables on subsets of reviews (e.g., most recent, most helpful) to address this.
  • Contextual Information: While the current metrics are effective, future research could explore incorporating more contextual information, such as product feature-level sentiment diversity or the relevance of discussions.
  • Prediction vs. Explanation: The current work focuses on explaining sales effects. Future research could leverage these metrics for sales prediction.
  • Broader WOM Contexts: The study focused on book reviews where both ratings and text were available. Future work could investigate the impact of sentiment divergence in settings where only textual WOM is present (e.g., message boards, blogs).

Summary updated

BibTeX
@article{zhang2012deciphering,
  author = {Zhu Zhang and Xin Li and Yubo Chen},
  title = {{Deciphering word-of-mouth in social media: Text-based metrics of consumer reviews}},
  journal = {ACM Trans. Manag. Inf. Syst},
  year = {2012},
  doi = {10.1145/2151163.2151168},
}
Figure