Positivity Bias in Customer Satisfaction Ratings

Kunwoo Park, Meeyoung Cha, Eunhee Rhim

TL;DR

Online customer ratings are notoriously biased toward the positive, but what about the "silent majority" who never leave a review? By applying deep learning to over 170,000 Samsung customer service chat logs, researchers found that while 80% of respondents leave positive feedback, more than half of those who remain silent are likely dissatisfied.

What Problem Does This Paper Address?

We’ve all seen the "J-shaped distribution" of online reviews: people tend to leave either 5 stars or 1 star, with a heavy lean toward the positive. This creates a positivity bias that makes it difficult for companies to understand true customer sentiment.

The challenge is that the vast majority of customers (nearly 84% in this study) never fill out the post-interaction survey. Are these "silent" customers happy, or are they so frustrated they’ve simply given up? This paper aims to bridge this gap by using the content and rhythm of chat conversations to predict the satisfaction scores of the non-respondents, revealing the hidden level of dissatisfaction in the system.

Methods

The researchers moved beyond simple keyword matching to capture the "dynamic flow" of a conversation. They compared traditional feature-based models against deep learning architectures:

  • Baselines: They used Valence (sentiment scores via the VADER lexicon) and N-grams (frequency of word sequences).
  • Deep Learning: They implemented Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). These models are better at handling "sequence dependency"—the way a customer’s mood evolves over the course of a 20-minute chat.
  • The "Time Gap" Innovation: Standard text analysis often ignores when things are said. The authors encoded the pauses between messages (e.g., Short_Speaker, Long_Speaker) as tokens within the text sequence. This allowed the models to learn if a long pause from an agent or a quick, curt response from a customer was a signal of impending dissatisfaction.

Datasets

The study utilized a proprietary dataset from Samsung’s live chat service (cell phone category) in the United States:

  • Total Sessions: 173,886 English chat sessions over one year.
  • Labeled Data: 16.22% of sessions (where customers provided a 1-5 star rating).
  • Unlabeled Data: 83.78% of sessions (the "silent" majority).
  • Validation: To ensure the model wasn't hallucinating, three human coders manually labeled a subset of the unrated chats to see if they agreed with the AI's predictions.

Key Insights and Ideas

  • Silence is not Golden: The core finding is the "Positivity Hypothesis." Customers who don't leave ratings are significantly more likely to have had a bad experience than those who do.
  • Context Trumps Keywords: Traditional sentiment analysis (Valence) had high precision but terrible recall. It could identify "angry" words, but it missed the subtle, apathetic, or frustrated tones that don't use "bad" words. The LSTM models, by looking at the whole sequence, were much more robust.
  • Temporal Cues are Features: The inclusion of time gaps (the -Time versions of the models) consistently improved performance. A "Long_Agent" pause (49+ seconds) often correlates with a drop in customer satisfaction, acting as a non-verbal cue that the model can "read."

Key Results

  • Model Performance: The LSTM-Time and GRU-Time models were the clear winners, achieving an F1 score of approximately 0.80, significantly outperforming the N-gram (0.73) and Valence (0.49) baselines.
  • The Bias Revealed:
    • Among those who actually took the survey, 79.7% gave positive or neutral ratings.
  • For the silent non-respondents, the model predicted that only 45.5% would have been positive or neutral.
  • In other words, 54.5% of the silent majority were likely dissatisfied, compared to only about 20% of the vocal group.
  • The Combined Reality: When the known and inferred ratings are combined, the "overwhelmingly positive" service record disappears, replaced by a much more balanced—and realistic—view of customer pain points.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Manual Validation Scale: The human-coding validation was performed on a relatively small sample (100 sessions).
  • Single Source: The data comes exclusively from Samsung's US-based support. The authors noted that survey response rates in other countries (like India or the UK) are even lower (1-3%), suggesting that cultural norms play a massive role in rating behavior.
  • Model Sophistication: While LSTMs are powerful, the authors suggest that future work could use Attention Mechanisms to pinpoint exactly which parts of a conversation (e.g., the greeting vs. the solution) most heavily influence the final rating.

Summary updated

Abstract

Customer ratings are valuable sources to understand their satisfaction and are critical for designing better customer experiences and recommendations. The majority of customers, however, do not respond to rating surveys, which makes the result less representative. To understand overall satisfaction, this paper aims to investigate how likely customers without responses had satisfactory experiences compared to those respondents. To infer customer satisfaction of such unlabeled sessions, we propose models using recurrent neural networks (RNNs) that learn continuous representations of unstructured text conversation. By analyzing online chat logs of over 170,000 sessions from Samsung's customer service department, we make a novel finding that while labeled sessions contributed by a small fraction of customers received overwhelmingly positive reviews, the majority of unlabeled sessions would have received lower ratings by customers. The data analytics presented in this paper not only have practical implications for helping detect dissatisfied customers on live chat services but also make theoretical contributions on discovering the level of biases in online rating platforms.
BibTeX
@inproceedings{park2018positivity,
  author = {Kunwoo Park and Meeyoung Cha and Eunhee Rhim},
  title = {{Positivity Bias in Customer Satisfaction Ratings}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3184558.3186579},
}

Loading related content...

Figure