Predicting Customer Satisfaction with Soft Labels for Ordinal Classification

Etienne Manderscheid, Matthias LeeACL 2023

TL;DR

This paper tackles the challenge of predicting customer satisfaction (CSAT) in call centers, where survey response rates are low and biased. They introduce a novel deep learning approach that uses "soft labels" and a binary classification strategy to accurately infer CSAT scores from call transcripts, enabling proactive customer service improvements.

What Problem Does This Paper Address?

Call centers rely on Customer Satisfaction (CSAT) surveys to gauge customer experience, typically on a 1-5 scale. However, a major hurdle is the extremely low response rate (often less than 8%), and the responses that are collected are heavily skewed. Customers with very positive or very negative experiences are far more likely to leave feedback, meaning the collected data doesn't represent the majority of customers. This data sparsity and bias leave call center managers and coaches with incomplete information, preventing them from identifying and addressing issues for customers who don't leave feedback but had a suboptimal experience. This can lead to missed opportunities for service recovery and overall customer experience improvement.

Methods

The authors propose a two-pronged approach to overcome the data limitations and meet strict latency requirements:

  1. Binary Classification + Fan-Out: Instead of directly predicting one of five CSAT classes, they first reframe the problem as a binary classification task (e.g., distinguishing between "low CSAT" and "not low CSAT"). During inference, they use a "fan-out" mechanism with temperature-scaled probabilities and learned thresholds to map the binary output back to the original five CSAT classes. This strategy is designed to prioritize precision for the critical low CSAT classes.
  2. Soft Labels: Traditional "hard labels" assign a single, definitive class to each data point. This paper introduces "soft labels," which are a form of label smoothing adapted for ordinal data. Instead of a strict 1 for the correct class and 0 for others, soft labels distribute probability across related classes. For CSAT, this means a customer with a "2" rating might have a soft label that assigns a high probability to class "2," but also some probability to "1" and "3," reflecting the ordinal nature of the scale. They use different smoothing parameters ($\alpha$) for different CSAT classes, giving more smoothing to the middle classes (3) and less to the extreme classes (1 and 5).

These techniques are applied to a Big Bird transformer model, chosen for its ability to handle long sequences with low latency.

Datasets

The primary dataset consists of call transcripts from contact center conversations. The labels are derived from customer CSAT survey responses, aggregated across multiple companies. The distribution of these labels is heavily skewed towards the extremes (1 and 5), as shown in Table 1. The authors also mention creating training, validation, and test sets, ensuring no overlap between them to prevent data contamination.

Key Insights and Ideas

  • The Power of Soft Labels for Ordinal Data: The ordinal nature of CSAT (where a '2' is closer to a '1' than a '3') is crucial. Soft labels effectively capture this nuance, leading to better performance than hard labels in their setup.
  • Binary Classification as a Precision Booster: Reframing the problem as binary classification, especially for the critical low CSAT scores, allows the model to focus on maximizing precision for these specific classes. The "fan-out" mechanism then intelligently reconstructs the five-class prediction.
  • Latency is King: The need for near real-time predictions (within 10 seconds) heavily influenced model architecture and processing choices, driving the use of Big Bird and transcript-only input.
  • Precision Over Recall for Low CSAT: For use cases like agent coaching, it's more important to be highly confident that a predicted low CSAT score is actually low, even if it means missing a few low CSAT calls (lower recall). This is why "precision*" (precision with a tolerance of 1) is a key metric.

Key Results

  • Soft Labels Outperform: Binary classification with soft labels achieved the best performance across most metrics, particularly for precision on classes 1 and 5.
  • High Precision for Critical Classes: The "binary soft labels" approach achieved a precision of 83.4% for Class 1 (the most dissatisfied customers), which is significantly higher than the other methods.
  • Overall Accuracy: The binary soft labels and 5-way classification methods were statistically tied in overall accuracy (around 90.6%), with binary hard labels slightly behind.
  • Production Ready: The developed model is deployed at scale, processing over 50 million calls and providing predictions within the required 10-second latency.

Limitations and Future Work

  • Transcript-Only Input: The model relies solely on call transcripts. This means it cannot infer information about call duration, hold times, or wait times unless explicitly mentioned by the customer. Future work could incorporate these additional metadata features.
  • Data Dependency: The model's accuracy is dependent on the CSAT survey data it's trained on. If a call center doesn't use the company's survey system, the model's performance might be impacted.
  • Real-time Prediction: While the current system provides predictions 10 seconds post-call, the authors suggest exploring continuous prediction throughout the call for more immediate interventions (like agent whispering). This would require managing cost and computational footprint, potentially with smaller gating models.
  • Potential Bias: The authors acknowledge the need to sample subpopulations and evaluate internally to ensure the model isn't biased against specific user groups.

Summary updated

BibTeX
@inproceedings{manderscheid2023predicting,
  author = {Etienne Manderscheid and Matthias Lee},
  title = {{Predicting Customer Satisfaction with Soft Labels for Ordinal Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure