Robust angry speech detection employing a TEO-based discriminative classifier combination

Wooil Kim, John H. L. HansenInterspeech 2009

TL;DR

This paper introduces a more accurate method for detecting angry speech by combining a novel Teager Energy Operator (TEO)-based feature with traditional Mel-Frequency Cepstral Coefficients (MFCCs). By using discriminative training and a smart combination strategy, they significantly improve angry speech detection accuracy, especially in challenging open-vocabulary and open-speaker scenarios.

What Problem Does This Paper Address?

Detecting specific emotions or states in speech, like anger, is crucial for improving human-computer interaction, particularly in applications like call centers. While existing methods can detect stressed speech, accurately identifying angry speech, especially when the vocabulary and speakers are unknown, remains a challenge. Previous approaches often relied on features that might not capture the subtle acoustic cues of anger effectively, or they struggled with variations in speech production. This paper aims to bridge this gap by developing a more robust and accurate angry speech detection system that can generalize across different speakers and words.

Methods

The core of the proposed method lies in a multi-pronged approach:

  • TEO-CB-Auto-Env Feature Extraction: This is a novel feature designed to capture excitation characteristics in speech. It involves using Gabor bandpass filters, segmenting the Teager Energy Operator (TEO) profile, and then computing the area under the autocorrelation envelope. This feature is particularly good at reflecting variations in vocal fold excitation, which are often altered during stressed speech.
  • Decorrelation (DCT): To improve the training ability of the TEO-based features, the authors apply a Discrete Cosine Transform (DCT). This process reduces the correlation between feature elements and the overall vector size, making the acoustic models more efficient to train.
  • Minimum Classification Error (MCE) Training: Instead of standard Maximum Likelihood Estimation (MLE), the authors employ MCE training. This discriminative training method directly optimizes for minimizing classification errors, leading to acoustic models that are better at distinguishing between different speech states (e.g., angry vs. neutral, loud, or Lombard speech).
  • Feature and Classifier Combination:
    • Feature Combination: The TEO-based features (including the decorrelated version, DTEO) are concatenated with traditional MFCC features. MFCCs are known for their effectiveness in capturing spectral envelope characteristics, and combining them with TEO features aims to leverage the strengths of both.
  • Classifier Combination: Separate classifiers are trained on TEO and MFCC features. Their likelihood scores are then combined using a weighted sum (controlled by a parameter $\alpha$) at the decision stage. This allows for a more flexible integration of information from different feature sets.

Datasets

The primary dataset used for evaluation is the Speech Under Simulated and Actual Stress (SUSAS) corpus.

  • SUSAS Corpus: This corpus contains over 16,000 isolated-word utterances from 32 speakers across five stress domains (talking styles, speech in noise, computer tasks, motion-fear tasks, and psychiatric analysis). It includes a common set of 35 aircraft communication words, making it suitable for open-vocabulary tasks. The authors specifically used data from 10 stressed styles (talking styles, single tracking task, and Lombard effect domains) for simulated stress, and actual speech under stress data from workload and motion-fear tasks.
  • Data Split: The experiments were conducted in an open-speaker and open-vocabulary setting, meaning speakers and words in the test set were not present in the training or development sets. They generated multiple combinations of word sets for training, development, and testing to ensure a comprehensive evaluation.

Key Insights and Ideas

  • TEO Features Capture Stress: The TEO-CB-Auto-Env feature is effective at capturing excitation variations, which are key indicators of stressed speech, including anger.
  • Decorrelation Boosts Trainability: Applying DCT to TEO features significantly improves model training ability by reducing feature redundancy. This is a crucial step for making these features more practical.
  • Discriminative Training is Key: MCE training consistently improves the discriminative power of acoustic models, leading to better separation between different speech classes compared to standard MLE.
  • Combination Strategies Work: Both feature and classifier combination approaches yield performance improvements over single features.
  • Classifier Combination Outperforms Feature Combination with MCE: While feature combination showed promise, classifier combination, especially when using MCE-trained models, proved more robust, particularly under the "sparse data" conditions of MCE training with high-dimensional features. This suggests that allowing classifiers to learn independently and then combining their decisions can be more effective.

Key Results

  • The proposed system, combining decorrelated TEO features, MCE training, and classifier combination, achieved up to 7.78% improvement in classification accuracy compared to using only the single TEO-based feature.
  • The decorrelated TEO feature (DTEO) generally outperformed the original TEO feature, demonstrating the benefit of the DCT decorrelation step.
  • Both feature and classifier combination approaches showed improvements over single features.
  • Classifier combination with MCE training achieved the best performance, outperforming MLE-based classifier combination and feature combination approaches. For example, the DTEO & MFCC classifier combination with MCE training achieved an average accuracy of 85.36% in the 4-class classification task.

Limitations and Future Work

  • "Sparse Data" Problem in MCE: The authors observed degraded performance when applying MCE training to feature combination with high-dimensional features (31 and 25 elements). They attribute this to a "sparse data" problem, where the development data might not sufficiently represent the feature space of the test data for MCE to effectively optimize. They suggest that a model combination approach might address this.
  • Future Work: The authors propose exploring potential speaker-dependent traits for the classifier combination factor ($\alpha$). This suggests that dynamically adjusting the weighting of TEO and MFCC contributions based on speaker characteristics could further enhance performance.

Summary updated

BibTeX
@inproceedings{kim2009robust,
  author = {Wooil Kim and John H. L. Hansen},
  title = {{Robust angry speech detection employing a TEO-based discriminative classifier combination}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure