Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations

Yelin Kim, Joshua Levy, Yang Liu

Abstract

For an interactive agent, such as task-oriented spoken dialog systems or chatbots, measuring and adapting to Customer Satisfaction (CSAT) is critical in order to understand user perception of an agent's behavior and increase user engagement and retention. However, an agent often relies on explicit customer feedback for measuring CSAT. Such explicit feedback may result in potential distraction to users and it can be challenging to capture continuously changing user's satisfaction. To address this challenge, we present a new approach to automatically estimate CSAT using acoustic and lexical information in the Alexa Prize Socialbot data. We first explore the relationship between CSAT and sentiment scores at both the utterance and conversation level. We then investigate static and temporal modeling methods that use estimated sentiment scores as a mid-level representation. The results show that the sentiment scores, particularly valence and satisfaction, are correlated with CSAT. We also demonstrate that our proposed temporal modeling approach for estimating CSAT achieves competitive performance, relative to static baselines as well as human performance. This work provides insights into open domain social conversations between real users and socialbots, and the use of both acoustic and lexical information for understanding the relationship between CSAT and sentiment scores.

Introduction

In the development of AI assistants, understanding user satisfaction is the primary metric for performance improvement. Traditionally, developers rely on explicit feedback prompts (e.g., "Rate your experience"), which are often intrusive and suffer from low response rates. While predicting CSAT is relatively straightforward for task-oriented bots, it is notoriously difficult for open-domain "socialbots" where goals are ill-defined. This research investigates whether satisfaction can be inferred directly from the user's tone and words, bypassing the need for intrusive surveys.

Methodology

The proposed system utilizes a two-phase architecture to estimate CSAT from raw conversation data:

  1. Utterance-Level Sentiment Extraction: Every turn in a conversation is processed through two paths. An Acoustic Path extracts 40-dimensional Log Mel-filter bank energies (LFBEs) via a 5-layer LSTM, while a Lexical Path uses 300-dimensional word2vec embeddings from ASR transcripts via a 1-layer LSTM. These are concatenated to predict three sentiment dimensions: Activation, Valence, and Satisfaction.
  2. Conversation-Level CSAT Estimation: The sequence of sentiment embeddings is fed into a regression model. We compared a static model ($\nu$-SVR) against a temporal model (BLSTM) designed to capture how sentiment evolves over the course of a chat.
Overview of the two-phase CSAT estimation architecture, showing the flow from utterances to sentiment embeddings and finally to CSAT estimation.

Experimental Setup

The study utilized two primary datasets from the Alexa Prize:

  • Unannotated Data: 6,308 conversations paired with real user-reported CSAT scores (1-5 stars), used for training the final estimator.
  • Annotated Data: A subset of 952 conversations manually labeled by human experts for sentiment dimensions to provide ground truth for the extraction phase.

Key Results and Analysis

The temporal BLSTM model demonstrated superior performance on the larger unannotated dataset, achieving a correlation of 0.2366—a 12.56% relative improvement over the static baseline. Notably, this automated model slightly outperformed human annotators (0.2244), who often struggled to predict the actual ratings given by users. This suggests that the model captures subtle acoustic and lexical cues that third-party observers might overlook.

Comparison of CSAT estimation performance (correlation) between human annotators and automated models (SVR and BLSTM) across annotated and unannotated datasets.

Analysis of sentiment dimensions revealed that Valence (positivity) and Satisfaction (frustration vs. delight) are significantly stronger predictors of CSAT than Activation. High activation can signal either extreme joy or extreme anger, making it a noisier indicator. Furthermore, the success of the BLSTM highlights that the temporal dynamics of a conversation—such as a poor start followed by a successful recovery—are critical to the final user rating.

Conclusion

This work demonstrates that automated CSAT estimation using sentiment as a mid-level representation is a viable alternative to explicit user feedback. While the BLSTM excels on large datasets, performance on smaller annotated sets suggests that data scarcity remains a challenge for complex temporal models. Future work will explore integrating features related to bot response quality, such as coherence and factual accuracy, to further refine satisfaction estimates.

Summary updated

Abstract

For an interactive agent, such as task-oriented spoken dialog systems or chatbots, measuring and adapting to Customer Satisfaction (CSAT) is critical in order to understand user perception of an agent's behavior and increase user engagement and retention. However, an agent often relies on explicit customer feedback for measuring CSAT. Such explicit feedback may result in potential distraction to users and it can be challenging to capture continuously changing user's satisfaction. To address this challenge, we present a new approach to automatically estimate CSAT using acoustic and lexical information in the Alexa Prize Socialbot data. We first explore the relationship between CSAT and sentiment scores at both the utterance and conversation level. We then investigate static and temporal modeling methods that use estimated sentiment scores as a mid-level representation. The results show that the sentiment scores, particularly valence and satisfaction, are correlated with CSAT. We also demonstrate that our proposed temporal modeling approach for estimating CSAT achieves competitive performance, relative to static baselines as well as human performance. This work provides insights into open domain social conversations between real users and socialbots, and the use of both acoustic and lexical information for understanding the relationship between CSAT and sentiment scores.
BibTeX
@misc{kim2020speech,
  author = {Yelin Kim and Joshua Levy and Yang Liu},
  title = {{Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations}},
  year = {2020},
  eprint = {2008.12376},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.12376},
}
Figure