Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion

Ziang Zhou, Yanze Xu, Shilei Zhang, Ming LiarXiv 2021 · arXiv

Abstract

This paper presents a robust system for detecting conversational escalation levels (low, medium, or high) from speech. By leveraging transfer learning from large-scale emotion recognition datasets and fusing acoustic features with textual embeddings, the authors achieved a significant performance boost over the standard baseline for the ComParE 2021 Escalation Sub-task, reaching an 81.5% Unweighted Average Recall (UAR).

Introduction

Detecting when a conversation is escalating—moving from a calm interaction to a heated conflict—is vital for customer service safety and public security in environments like airports or train stations. However, building machine learning models for this task is challenging due to data scarcity, environmental noise in real-world recordings, and the inherent complexity of escalation, which is expressed through both prosody (tone, volume) and lexical content (keywords, sentiment).

Methodology

The researchers developed a multi-stage pipeline to address these challenges, focusing on a multimodal approach that combines acoustic and textual information.

The proposed system architecture for escalation detection, featuring acoustic and textual feature extraction followed by information fusion and SVM classification.

Voice Activity Detection (VAD)

To ensure the model focused only on human vocalizations, the WebRTC-VAD tool was used to strip away non-speech segments. This simple cleaning step proved essential, improving UAR from 67.5% to 71.0%.

Transfer Learning and Feature Extraction

To overcome the small size of the escalation dataset (~400 training samples), the authors employed transfer learning. A ResNet-18 model was first trained on four large-scale emotional audio datasets (RAVDESS, CREMA-D, SAVEE, and TESS), totaling over 12,000 labeled clips. The hypothesis was that the acoustic patterns of anger or frustration are highly correlated with escalation.

For feature extraction:

  • Acoustic Features: Mel Frequency Cepstral Coefficients (MFCCs) were processed through the pre-trained ResNet-18 to generate a 512-dimensional representation.
  • Textual Features: A multilingual Sentence-BERT (SBERT) model extracted 768-dimensional embeddings from Dutch transcriptions (translated to English for optimized performance).

Information Fusion and Classification

The system experimented with both early fusion (concatenation) and late fusion (voting). The final classification was performed using Support Vector Machines (SVM), which are often more robust than standard softmax layers for small-scale datasets.

Key Results and Insights

The proposed system achieved a top performance of 81.5% UAR on the development set, representing a 12.8% absolute improvement over the 72.2% baseline provided by the ComParE 2021 organizers.

Comparison of different fusion approaches, showing that the Voting mechanism achieved the highest performance across all metrics.

The Dominance of Acoustic Features

One of the most significant findings was the effectiveness of the pre-trained acoustic model. The ResNet-18 alone reached 81.0% UAR. Interestingly, adding textual embeddings provided only marginal improvements. This suggests that in short 5-second clips, how something is said (prosody) carries more weight than the specific words used.

Late Fusion Performance

As shown in the results, a late fusion "Voting" ensemble of different models (including MFCC-based and Log-filterbank-based variants) proved more stable and accurate than simple concatenation or mean-based fusion.

Limitations and Future Work

While successful, the authors identified several areas for improvement. The Dutch-to-English translation required for SBERT may have introduced errors, potentially limiting the impact of textual features. Additionally, standard denoising algorithms were found to degrade performance, likely by altering emotional characteristics in the speech signal. Future research will focus on "emotion-preserving" noise reduction and more sophisticated neural-network-based VAD to handle complex conversational overlaps.

Summary updated

BibTeX
@misc{zhou2021detecting,
  author = {Ziang Zhou and Yanze Xu and Shilei Zhang and Ming Li},
  title = {{Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion}},
  year = {2021},
  eprint = {2104.06004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.06004},
}
Figure