Predicting Self-reported Customer Satisfaction of Interactions with a Corporate Call Center

Joseph Bockhorst, Shi Yu, Luisa F. Polanía, Glenn FungECML/PKDD 2017 · DOI

TL;DR

This paper introduces a novel system for predicting customer satisfaction after phone calls by combining speech-to-text transcripts, call metadata, and customer data. Their "rank score + isotonic regression" (RS+IR) approach effectively handles the ordinal and skewed nature of satisfaction scores, outperforming standard regression and classification methods. This system is now deployed in a production environment, impacting tens of thousands of calls daily.

What Problem Does This Paper Address?

In today's competitive landscape, customer satisfaction is paramount for business success. Companies rely on customer surveys to gauge satisfaction after interactions, particularly with call centers. However, these surveys are expensive, leading to small sample sizes and potentially unreliable performance metrics for customer representatives (CRs). This variability can misrepresent CR performance and prevent timely interventions to improve customer experience. The challenge is to accurately predict customer satisfaction for every call, enabling proactive customer service and targeted training.

Methods

The authors propose a two-stage modeling workflow to address the challenges of ordinal and skewed satisfaction scores:

  • Rank Score Model: Instead of directly predicting the satisfaction score, they first train a linear ranking model. This model learns to order pairs of calls based on their satisfaction scores. This is achieved by creating an auxiliary training set of example pairs and training a binary classifier to predict which example in a pair has a higher satisfaction score. This approach is more robust to the skewed distribution of scores.
  • Isotonic Regression Model: The output of the rank score model (a rank score) is then fed into a non-decreasing, one-dimensional isotonic regression model. This model maps the rank scores to actual satisfaction scores. Isotonic regression is chosen because it preserves the ordinal nature of the data and is well-suited for fitting to smoothed sequences of satisfaction scores.

This combined approach, termed RS+IR, is designed to be more accurate than standard regression methods that assume interval or ratio scales and can be generalized to other ordinal prediction problems.

Beyond the core modeling, the system also involves:

  • Data Integration: Combining data from speech-to-text transcripts, call logs (duration, wait times), and customer/policy databases.
  • Feature Engineering: Extracting features from call transcripts (e.g., sentiment trends over time) and call metadata. They specifically highlight the use of temporally segmented features, recognizing that different parts of a call (e.g., problem explanation vs. resolution) might hold different predictive power.

Datasets

The study uses a dataset of 8,726 incoming phone calls from a large US insurance company.

  • Training Set: 75% of the data (6,108 calls) was used for training.
  • Test Set: The remaining 25% (2,618 calls) was used for evaluation.
  • Data Sources: Call logs, historical survey scores, customer and policy databases, and speech-to-text transcriptions of calls.

No new benchmark datasets were introduced; the work focuses on applying their method to real-world company data.

Key Insights and Ideas

  • Ordinality Matters: Directly applying standard regression to ordinal data (like satisfaction scores) can be suboptimal. Treating satisfaction as a ranking problem first, then mapping to scores, proves more effective.
  • Skewed Data is a Challenge, but Solvable: The highly skewed distribution of satisfaction scores (most customers are satisfied) is a significant modeling hurdle. The RS+IR approach, by focusing on relative ordering, mitigates this issue.
  • Temporal Context is Crucial: Analyzing call segments (e.g., sentiment trends in the first quarter vs. the last) provides richer signals than treating the entire call transcript as a single block of text.
  • Group-Level Predictions are Robust: While individual call predictions are valuable, the system's accuracy at the group level (e.g., for a specific call queue or CR) is particularly strong, especially for larger groups. This suggests reliability for operational reporting.

Key Results

  • Individual Predictions: The RS+IR model outperformed standard regression methods (Ridge, Lasso, Random Forest) and a linear SVM in terms of Pearson correlation, Spearman correlation, and Mean Absolute Error (MAE) on the test set.
  • RS+IR achieved a Pearson correlation of 0.312 and MAE of 0.784.
  • It also showed superior performance in classification tasks (AUC) compared to linear SVM, especially for identifying potentially dissatisfied customers.
  • Group Predictions:
    • The system's group error profiles were found to be similar to natural groupings by topic.
  • For topic groups with at least 50 calls, 93.75% of their errors fell within the 95% confidence bands derived from random group errors, indicating good predictive consistency.
  • A Pearson correlation of 0.73 was observed between predicted and actual group means for topic groups.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the RS+IR model is more accurate than standard regression, it's still a challenge to achieve very high prediction accuracy for individual calls due to the inherent subjectivity and variability of customer responses.
  • The authors note that for classification tasks (e.g., identifying customers below a certain satisfaction threshold), dedicated classification models might be more appropriate, although their RS+IR model also performed well in this regard.
  • Future Work:
    • The paper suggests that the RS+IR approach is generalizable to other ordinal prediction problems.
  • The system is already deployed and generating reports, implying ongoing monitoring and potential for further refinement based on real-world performance.

Summary updated

BibTeX
@inproceedings{bockhorst2017predicting,
  author = {Joseph Bockhorst and Shi Yu and Luisa F. Polanía and Glenn Fung},
  title = {{Predicting Self-reported Customer Satisfaction of Interactions with a Corporate Call Center}},
  booktitle = {ECML/PKDD},
  year = {2017},
  doi = {10.1007/978-3-319-71273-4\_15},
}

Loading related content...

Figure