Abstract
Customer ratings are valuable sources to understand their satisfaction and are critical for designing better customer experiences and recommendations. The majority of customers, however, do not respond to rating surveys, which makes the result less representative. To understand overall satisfaction, this paper aims to investigate how likely customers without responses had satisfactory experiences compared to those respondents. To infer customer satisfaction of such unlabeled sessions, we propose models using recurrent neural networks (RNNs) that learn continuous representations of unstructured text conversation. By analyzing online chat logs of over 170,000 sessions from Samsung's customer service department, we make a novel finding that while labeled sessions contributed by a small fraction of customers received overwhelmingly positive reviews, the majority of unlabeled sessions would have received lower ratings by customers. The data analytics presented in this paper not only have practical implications for helping detect dissatisfied customers on live chat services but also make theoretical contributions on discovering the level of biases in online rating platforms.
Introduction: The "Silent Majority" Problem
Online customer reviews often exhibit a "J-shaped distribution," where feedback is polarized toward extreme positive or negative ratings, with a significant lean toward the positive. This creates a systemic positivity bias that obscures true customer sentiment. In this study of Samsung’s live chat service, nearly 84% of customers opted out of post-interaction surveys. The central question is whether these "silent" customers are satisfied or if their silence masks frustration. By predicting the satisfaction scores of non-respondents using chat content and temporal dynamics, this research reveals the hidden level of dissatisfaction within the service ecosystem.
Methodology: Capturing Dynamic Flow
To move beyond simple keyword-based sentiment analysis, the researchers employed deep learning architectures capable of capturing the "dynamic flow" of a conversation. While traditional models like N-grams and Valence (using the VADER lexicon) provide a baseline, they often fail to account for sequence dependency—how a customer's mood evolves over a 20-minute interaction.
The study utilized Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). A key innovation was the inclusion of "Time Gap" tokens. By encoding the duration of pauses between messages (e.g., SHORT_CUSTOMER, LONG_AGENT), the models could learn to interpret non-verbal cues. For instance, a long pause from an agent often correlates with a drop in customer satisfaction.
Data and Model Performance
The dataset comprised 173,886 English chat sessions from Samsung’s US cell phone support. Only 16.22% of these sessions were labeled with a 1-5 star rating. The LSTM-Time and GRU-Time models emerged as the most effective, achieving an F1 score of approximately 0.80, significantly outperforming traditional N-gram (0.73) and Valence (0.49) methods. The inclusion of temporal features consistently improved performance, proving that when something is said is as important as what is said.
Key Results: Silence is Not Golden
The core finding supports the "Positivity Hypothesis": customers who do not leave ratings are significantly more likely to have had a negative experience.
- Labeled Data: Among survey respondents, 79.7% provided positive or neutral ratings.
- Unlabeled Data: The model predicted that only 45.5% of the silent majority would have provided positive or neutral feedback.
- The Reality Gap: Over 54% of the silent majority were likely dissatisfied, compared to only 20% of the vocal group.
When these inferred ratings are combined with the known data, the service record shifts from "overwhelmingly positive" to a more balanced and realistic view of customer pain points.
Conclusion and Future Work
This research demonstrates that relying solely on voluntary ratings leads to a significant underestimation of customer dissatisfaction. While LSTMs with temporal tokens provide a robust predictive framework, future work could incorporate attention mechanisms to identify specific conversational triggers that influence ratings. Additionally, expanding the study to different geographic regions could reveal how cultural norms influence the propensity to leave feedback, as response rates vary significantly across global markets.
Summary updated