Abstract
This paper presents a robust system for detecting conversational escalation levels (low, medium, or high) from speech. By leveraging transfer learning from large-scale emotion recognition datasets and fusing acoustic features with textual embeddings, the authors achieved a significant performance boost over the standard baseline for the ComParE 2021 Escalation Sub-task, reaching an 81.5% Unweighted Average Recall (UAR).
Introduction
Detecting when a conversation is escalating—moving from a calm interaction to a heated conflict—is vital for customer service safety and public security in environments like airports or train stations. However, building machine learning models for this task is challenging due to data scarcity, environmental noise in real-world recordings, and the inherent complexity of escalation, which is expressed through both prosody (tone, volume) and lexical content (keywords, sentiment).
Methodology
The researchers developed a multi-stage pipeline to address these challenges, focusing on a multimodal approach that combines acoustic and textual information.
Voice Activity Detection (VAD)
To ensure the model focused only on human vocalizations, the WebRTC-VAD tool was used to strip away non-speech segments. This simple cleaning step proved essential, improving UAR from 67.5% to 71.0%.
Transfer Learning and Feature Extraction
To overcome the small size of the escalation dataset (~400 training samples), the authors employed transfer learning. A ResNet-18 model was first trained on four large-scale emotional audio datasets (RAVDESS, CREMA-D, SAVEE, and TESS), totaling over 12,000 labeled clips. The hypothesis was that the acoustic patterns of anger or frustration are highly correlated with escalation.
For feature extraction:
- Acoustic Features: Mel Frequency Cepstral Coefficients (MFCCs) were processed through the pre-trained ResNet-18 to generate a 512-dimensional representation.
- Textual Features: A multilingual Sentence-BERT (SBERT) model extracted 768-dimensional embeddings from Dutch transcriptions (translated to English for optimized performance).
Information Fusion and Classification
The system experimented with both early fusion (concatenation) and late fusion (voting). The final classification was performed using Support Vector Machines (SVM), which are often more robust than standard softmax layers for small-scale datasets.
Key Results and Insights
The proposed system achieved a top performance of 81.5% UAR on the development set, representing a 12.8% absolute improvement over the 72.2% baseline provided by the ComParE 2021 organizers.
The Dominance of Acoustic Features
One of the most significant findings was the effectiveness of the pre-trained acoustic model. The ResNet-18 alone reached 81.0% UAR. Interestingly, adding textual embeddings provided only marginal improvements. This suggests that in short 5-second clips, how something is said (prosody) carries more weight than the specific words used.
Late Fusion Performance
As shown in the results, a late fusion "Voting" ensemble of different models (including MFCC-based and Log-filterbank-based variants) proved more stable and accurate than simple concatenation or mean-based fusion.
Limitations and Future Work
While successful, the authors identified several areas for improvement. The Dutch-to-English translation required for SBERT may have introduced errors, potentially limiting the impact of textual features. Additionally, standard denoising algorithms were found to degrade performance, likely by altering emotional characteristics in the speech signal. Future research will focus on "emotion-preserving" noise reduction and more sophisticated neural-network-based VAD to handle complex conversational overlaps.
Summary updated