Multi-Grained Role Labeling Based on Multi-Modality Information for Real Customer Service Telephone Conversation

Weizhi Ma, Min Zhang, Yiqun Liu, Shaoping Ma

Introduction

Analyzing customer service call center data offers significant business intelligence, yet it presents unique technical challenges. Most recordings are single-stream, blending the voices of agents and customers into one track. Furthermore, unlike traditional speaker recognition, call centers involve thousands of agents and millions of unique customers, making pre-trained voiceprints for every individual impractical. Existing methods relying solely on acoustic features often fail due to poor line quality or similar vocal profiles. This paper, presented at IJCAI-16, proposes a multi-modal framework that combines acoustic signals with semantic text transcripts to accurately distinguish between speaker roles without prior identity knowledge.

Methodology: The Four-Phase Framework

The researchers developed a multi-grained approach to transition from raw audio to labeled roles. This framework integrates acoustic and textual features across four distinct phases to move from coarse diarization to fine-grained sentence labeling.

The four-phase multi-modal framework for role labeling.
  1. Speaker Diarization (Acoustic): Using the Bayesian Information Criterion (BIC), the system detects "change points" in the audio to group segments into two clusters. This identifies that two people are speaking but does not yet assign roles.
  2. Automatic Speech Recognition (ASR): Audio segments are processed through a Deep Neural Network-Hidden Markov Model (DNN-HMM) to generate text transcripts.
  3. Dialog-Level Role Labeling (Multi-Modal): This phase performs the initial role mapping. By analyzing the text from both clusters using a "bag-of-words" approach, a Decision Tree classifier identifies which cluster corresponds to the "Agent" versus the "User" based on typical professional vocabulary.
  4. Sentence-Level Role Correction (Textual): To correct diarization errors where segments might be misclustered, the system employs Relative Entropy to identify role-specific keywords (e.g., "account balance"). Logistic Regression then re-labels individual sentences if the textual model shows high confidence.

Datasets and Evaluation

The framework was evaluated on two real-world Chinese-language datasets:

  • MSDataset (Mobile Service): 34 conversations containing over 2,000 sentences.
  • BKDataset (Bank Service): 85,336 conversations, with 1,000 manually labeled for evaluation.

Key Results and Insights

The integration of textual features significantly enhances performance over acoustic-only methods. The system achieved over 99% accuracy in dialog-level role identification (AccuD) across both datasets. At the sentence level (AccuS), the multi-modal approach reached 90.4% accuracy, an 11.9% improvement over the 78.5% achieved by acoustic features alone.

Accuracy improvement on the MSDataset as a function of the correction threshold $\tau$.

A critical insight is the use of Relative Entropy to filter out common words and isolate "discriminative" jargon. Furthermore, the Confidence-Based Correction mechanism ensures that the system only overrides initial labels when the textual model is highly certain (using a threshold $\tau$), preventing the introduction of new errors. The model also demonstrated domain independence, with the bank-trained model performing nearly as well on mobile service data (97.1% AccuD).

Conclusion

By treating text as the "ground truth" for roles, this framework overcomes the limitations of purely acoustic diarization. While challenges remain regarding initial segmentation errors and cultural speech variations, this multi-modal approach provides a robust foundation for downstream tasks like automated satisfaction evaluation and real-time topic identification.

Summary updated

Abstract

Large-scale customer service call records include lots of valuable information for business intelligence. However, the analysis of those records has not utilized in the big data era before. There are two fundamental problems before mining and analyses: 1) The telephone conversation is mixed with words of agents and users which have to be recognized before analysis; 2) The speakers in conversation are not in a pre-defined set. These problems are new challenges which have not been well studied in the previous work. In this paper, we propose a four-phase framework for role labeling in real customer service telephone conversation, with the benefit of integrating multi-modality features, i.e., both low-level acoustic features and semantic-level textual features. Firstly, we conduct ΔBayesian Information Criterion (ΔBIC) based speaker diarization to get two segments clusters from an audio stream. Secondly, the segments are transferred into text in an Automatic Speech Recognition (ASR) phase with a deep learning model DNN-HMM. Thirdly, by intergrating acoustic and textual features, dialog level role labeling is proposed to map the two clusters into the agent and the user. Finally, sentence level role correction is designed in order to label results correctly in a fine-grained notion, which reduces the errors made in previous phases. The proposed framework is tested on two real datasets: mobile and bank customer service calls datasets. The precision of dialog level labeling is over 99.0%. On the sentense level, the accuracy of labeling reaches 90.4%, greatly outperforming traditional acoustic features based method which achieves only 78.5% in accuracy.
BibTeX
@inproceedings{ma2016multi,
  author = {Weizhi Ma and Min Zhang and Yiqun Liu and Shaoping Ma},
  title = {{Multi-Grained Role Labeling Based on Multi-Modality Information for Real Customer Service Telephone Conversation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure