Automatic Speech Feature Learning for Continuous Prediction of Customer Satisfaction in Contact Center Phone Calls

Carlos Segura, Daniel Balcells, Martí Umbert, Javier Arias, Jordi LuqueIberSPEECH 2016 · DOI

TL;DR

This paper introduces a novel end-to-end Convolutional Neural Network (CNN) architecture that learns meaningful speech features directly from raw audio for predicting customer satisfaction in contact centers. The approach demonstrates successful feature transfer from a conflict detection task, achieving comparable results to traditional Mel-scale features and outperforming them in terms of computational cost.

What Problem Does This Paper Address?

Understanding customer satisfaction in contact center phone calls is crucial for improving service quality. Traditionally, this has involved a two-step process: extracting hand-crafted audio features and then feeding them into a machine learning model. This approach requires significant domain expertise and can be suboptimal. The paper addresses the challenge of automatically learning robust and generalizable speech features directly from raw audio, bypassing the need for manual feature engineering and enabling more efficient and potentially more accurate prediction of customer satisfaction.

Methods

The core of the paper is a deep Convolutional Neural Network (CNN) architecture designed for end-to-end learning from raw audio signals. The network comprises six layers: four convolutional layers alternating with pooling layers, followed by two dense layers. The convolutional layers are responsible for extracting local temporal features, while pooling layers reduce dimensionality.

Key methodological aspects include:

  • Raw Audio Input: The CNN takes raw audio samples as input, avoiding traditional front-end processing like Mel-filter banks.
  • End-to-End Learning: The entire network, from feature extraction to prediction, is trained jointly.
  • Feature Transfer: The authors investigate transferring learned features from a conflict detection task (trained on French political debates) to the customer satisfaction prediction task (using Spanish call center data). This involves freezing the convolutional layers and retraining only the dense layers.
  • Training Strategy: The network is trained using stochastic gradient descent (SGD) with mini-batches. For the satisfaction prediction task, they compare training from scratch with transfer learning.

Datasets

  • SSPNet Conflict Corpus (SC2): This dataset, used for initial training and feature transfer, consists of 45 French political debates, annotated with a continuous conflict score. For the conflict detection task, a binary classification (high/low conflict) was created.
  • Call Center Data: This dataset comprises 19,360 inbound Spanish phone calls from a Latin American country. Customer satisfaction was self-reported on a 1-5 scale. For the satisfaction prediction task, a binary classification (high satisfaction ≥ 4 vs. low satisfaction ≤ 2) was created.

Key Insights and Ideas

  • Raw Audio is Sufficient: The CNN can effectively learn salient features directly from raw audio, achieving performance comparable to systems using traditional Mel-scale filter banks. This significantly simplifies the pipeline and reduces reliance on expert-designed features.
  • Feature Transferability: Learned features from a conflict detection task on one language (French) and domain (political debates) can be successfully transferred to a different task (customer satisfaction) in another language (Spanish) and domain (call centers). This highlights the generalizability of the learned representations.
  • Computational Efficiency: While the raw-audio front-end has a higher computational cost during training, it can lead to speedups during inference due to GPU optimization.
  • Domain Agnosticism: The learned features appear to be robust and meaningful across different domains and tasks, suggesting a potential for broader applicability.

Key Results

  • Conflict Detection: The raw-audio CNN achieved comparable results to spectrogram-based CNNs and traditional methods on the SSPNet Conflict test database, with Unweighted Average Recall (UAR) and Correlation Coefficient (CC) scores close to the baseline.
  • Satisfaction Prediction:
    • When trained from scratch, the raw-audio CNN achieved an AUC of 0.60 for satisfaction prediction using median aggregation.
  • With transfer learning from the conflict detection task, the AUC was 0.584. While slightly lower, this demonstrates the effectiveness of transfer learning.
  • The paper also found that using longer audio segments (up to 4 minutes) generally led to better performance, but even the last 30 seconds provided reasonable results.

Limitations and Future Work

  • Downsampling: The original SC2 corpus was downsampled to 8kHz to match the telephone channel bandwidth and facilitate training. This might lead to a loss of information.
  • Sampling Procedure: The sampling of 3-second excerpts from the SC2 corpus might include non-speech segments, which could introduce noise. However, the authors suggest that mini-batch size and training epochs mitigate this.
  • Limited Tuning: The authors acknowledge that they did not exhaustively tune their CNN architecture for optimal performance on the conflict detection task.
  • Future Work: The paper suggests exploring the learned features further and applying them to other tasks and languages. Investigating different aggregation strategies for continuous satisfaction scores and exploring more sophisticated transfer learning techniques are also potential avenues.

Summary updated

BibTeX
@inproceedings{segura2016automatic,
  author = {Carlos Segura and Daniel Balcells and Martí Umbert and Javier Arias and Jordi Luque},
  title = {{Automatic Speech Feature Learning for Continuous Prediction of Customer Satisfaction in Contact Center Phone Calls}},
  booktitle = {IberSPEECH},
  year = {2016},
  doi = {10.1007/978-3-319-49169-1\_25},
}
Figure