A Deep Learning System for Sentiment Analysis of Service Calls

Yanan JiaACL 2020

TL;DR

This paper introduces a novel deep learning system for sentiment analysis of service calls, focusing on real-world, noisy data. By combining acoustic and textual information, the system achieves improved accuracy in identifying negative customer sentiment, crucial for enhancing customer service and human-AI interactions.

What Problem Does This Paper Address?

Accurately understanding customer sentiment is vital for improving customer service and building more intelligent AI systems, especially in the context of human-robot interaction (HRI). While sentiment analysis has been extensively studied for text and monologues, real-world service calls present unique challenges: they are noisy, involve multi-party conversations, and often contain subtle emotional cues in vocal intonation that are missed by text-only analysis. Existing research often relies on clean, laboratory-recorded data or focuses solely on text. This paper aims to bridge this gap by developing a system that can effectively analyze sentiment in challenging, real-world service call environments, specifically focusing on identifying negative customer sentiment.

Methods

The paper proposes a bimodal sentiment analysis pipeline that leverages both textual and acoustic features from service calls.

  • Data Preparation: A crucial first step involves robust data preparation, including Voice Activity Detection (VAD) to remove silence, Automatic Speech Recognition (ASR) for transcription, and Automatic Speaker Diarization (ASD) to distinguish between Customer Service Representatives (CSRs) and customers. This process segments calls into utterances.
  • Textual Sentiment Analysis (CT): A combination of approaches is used. Initially, rule-based methods like VADER and sentiment APIs (AWSSA, GoogleSA) are employed for quick annotation. Later, more sophisticated machine learning models like Support Vector Machines (SVM), Long Short-Term Memory (LSTM), and Bidirectional LSTM (BLSTM) are trained on textual data.
  • Acoustic Sentiment Analysis (CA): Acoustic features are engineered based on human perception, focusing on loudness, sharpness, and speaking rate. These features are then aggregated over time windows and used to train classifiers like Elastic-Net, KNN, RF, and GMM.
  • Semi-supervised Learning Annotation Pipeline: To address the challenge of creating large annotated datasets, a semi-supervised learning pipeline is developed. This pipeline iteratively combines machine learning predictions with human correction, progressively improving the accuracy of both textual and acoustic classifiers.
  • Bimodal Fusion: The core novelty lies in the fusion of textual and acoustic modalities. The paper explores decision-level fusion, where the outputs of individual text and audio classifiers are combined. Two specific fusion strategies, Fus1 and Fus2, are proposed, with Fus2 showing particular promise by reclassifying ambiguous linguistic utterances based on audio cues.

Datasets

The primary dataset used is the Benefits Service Call Dataset (BSCD), a newly introduced benchmark consisting of 500 real-world service calls from a healthcare benefits call center. These calls are characterized by ambient noise and diverse communication patterns, making them more challenging than typical laboratory datasets. The dataset was pre-processed and annotated for sentiment. Additionally, 18,705 cleaned text chat data segments were incorporated into the training data to augment the textual information.

Key Insights and Ideas

  • Real-world data is messy but valuable: The BSCD dataset highlights the challenges of noisy, multi-party conversations but also underscores their importance for developing robust sentiment analysis systems.
  • Acoustic cues are critical for subtle sentiment: While text often carries the primary sentiment information, acoustic features (like tone, pitch, and speaking rate) are essential for disambiguating subtle emotional shifts and understanding the speaker's true intention, especially in borderline cases.
  • Semi-supervised learning is key for annotation: The proposed semi-supervised annotation pipeline significantly reduces the human effort required to label large datasets, making it more feasible to train complex models on real-world data.
  • Fusion improves performance: Combining textual and acoustic information, particularly through the Fus2 fusion strategy, leads to a statistically significant improvement in sentiment classification accuracy compared to using either modality alone.

Key Results

  • The proposed bimodal system (Fus2) achieved a 2% improvement in weighted F1-score over the unimodal text system.
  • McNemar's test confirmed a statistically significant improvement in accuracy when incorporating acoustic features using the Fus2 approach (p < 0.001).
  • The study found that acoustic modality alone achieved a weighted F1-score of 0.584, while text-based classifiers (SVM, LSTM, BLSTM) achieved scores around 0.718-0.719.
  • The temporal sentiment patterns revealed that customer negative scores often correlate with CSR negative scores, suggesting that sentiment can be influenced by interaction dynamics.

Limitations and Future Work

  • ASR and ASD errors: The system's reliance on ASR and ASD outputs means that errors in transcription or speaker identification can impact sentiment analysis accuracy.
  • Segment discarding: The semi-supervised annotation pipeline discards a significant portion of call segments (due to noise, silence, or ambiguity), limiting the ability to model inter-utterance dependencies within the current framework.
  • Future Work: The authors suggest exploring larger datasets, investigating more acoustic features, and improving the decision-level fusion process. They also highlight the need to address interdependent modeling of utterances within conversations as a future research direction.

Summary updated

BibTeX
@inproceedings{jia2020deep,
  author = {Yanan Jia},
  title = {{A Deep Learning System for Sentiment Analysis of Service Calls}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure