Introduction
Analyzing customer service call center data offers significant business intelligence, yet it presents unique technical challenges. Most recordings are single-stream, blending the voices of agents and customers into one track. Furthermore, unlike traditional speaker recognition, call centers involve thousands of agents and millions of unique customers, making pre-trained voiceprints for every individual impractical. Existing methods relying solely on acoustic features often fail due to poor line quality or similar vocal profiles. This paper, presented at IJCAI-16, proposes a multi-modal framework that combines acoustic signals with semantic text transcripts to accurately distinguish between speaker roles without prior identity knowledge.
Methodology: The Four-Phase Framework
The researchers developed a multi-grained approach to transition from raw audio to labeled roles. This framework integrates acoustic and textual features across four distinct phases to move from coarse diarization to fine-grained sentence labeling.
- Speaker Diarization (Acoustic): Using the Bayesian Information Criterion (BIC), the system detects "change points" in the audio to group segments into two clusters. This identifies that two people are speaking but does not yet assign roles.
- Automatic Speech Recognition (ASR): Audio segments are processed through a Deep Neural Network-Hidden Markov Model (DNN-HMM) to generate text transcripts.
- Dialog-Level Role Labeling (Multi-Modal): This phase performs the initial role mapping. By analyzing the text from both clusters using a "bag-of-words" approach, a Decision Tree classifier identifies which cluster corresponds to the "Agent" versus the "User" based on typical professional vocabulary.
- Sentence-Level Role Correction (Textual): To correct diarization errors where segments might be misclustered, the system employs Relative Entropy to identify role-specific keywords (e.g., "account balance"). Logistic Regression then re-labels individual sentences if the textual model shows high confidence.
Datasets and Evaluation
The framework was evaluated on two real-world Chinese-language datasets:
- MSDataset (Mobile Service): 34 conversations containing over 2,000 sentences.
- BKDataset (Bank Service): 85,336 conversations, with 1,000 manually labeled for evaluation.
Key Results and Insights
The integration of textual features significantly enhances performance over acoustic-only methods. The system achieved over 99% accuracy in dialog-level role identification (AccuD) across both datasets. At the sentence level (AccuS), the multi-modal approach reached 90.4% accuracy, an 11.9% improvement over the 78.5% achieved by acoustic features alone.
A critical insight is the use of Relative Entropy to filter out common words and isolate "discriminative" jargon. Furthermore, the Confidence-Based Correction mechanism ensures that the system only overrides initial labels when the textual model is highly certain (using a threshold $\tau$), preventing the introduction of new errors. The model also demonstrated domain independence, with the bank-trained model performing nearly as well on mobile service data (97.1% AccuD).
Conclusion
By treating text as the "ground truth" for roles, this framework overcomes the limitations of purely acoustic diarization. While challenges remain regarding initial segmentation errors and cultural speech variations, this multi-modal approach provides a robust foundation for downstream tasks like automated satisfaction evaluation and real-time topic identification.
Summary updated