Speaker dependent emotion recognition using prosodic supervectors

Ignacio Lopez-Moreno, Carlos Ortego-Resa, Joaquin Gonzalez-Rodriguez, Daniel RamosInterspeech 2009

TL;DR

This paper introduces a novel approach for recognizing emotions in speech by modeling prosodic features using Gaussian Mixture Models (GMMs) and classifying these models with Support Vector Machines (SVMs). Their method, "prosodic GMM-SVM," significantly improves emotion detection accuracy, especially when combined with a baseline system, by better capturing the statistical distribution of prosodic features.

What Problem Does This Paper Address?

Recognizing emotions from speech is a challenging but increasingly important task for human-machine interaction. The primary hurdles are: 1) speaker inter-variability, where different people express emotions differently, making it hard for models trained on one speaker to generalize to another, and 2) the ambiguity of emotions, where boundaries between emotional states can be fuzzy, and multiple emotions might co-exist within an utterance. This paper aims to tackle these challenges by developing a more robust method for modeling prosodic cues that convey emotional states.

Methods

The core of the proposed method, "prosodic GMM-SVM," involves:

  • Prosodic Feature Extraction: Short-term prosodic features like energy, log pitch, and their velocities are extracted from speech segments. These are then normalized.
  • GMM Supervector Modeling: For each utterance, a Gaussian Mixture Model (GMM) is trained to capture the statistical distribution of these prosodic features. Crucially, instead of just using summary statistics, the paper models the entire probability density function (pdf). The "supervector" is formed by concatenating the mean vectors of the GMM components. This supervector effectively summarizes the prosodic characteristics of the entire utterance in a high-dimensional space.
  • MAP Adaptation: The GMMs are adapted using Maximum A Posteriori (MAP) adaptation from a Universal Background Model (UBM). The UBM is trained on a large dataset and represents general, emotion-independent prosodic distributions, helping to mitigate speaker variability.
  • SVM Classification: A Support Vector Machine (SVM) is then trained to classify these GMM supervectors, distinguishing between different emotional states.

The paper also compares this to a baseline approach that extracts a fixed set of statistical coefficients (e.g., mean, max, min, standard deviation) from prosodic streams and classifies these directly with an SVM. Finally, they explore subsystem fusion, combining the scores from their proposed GMM-SVM system and the baseline system to further enhance performance.

Datasets

The experiments were conducted using the SUSAS (Speech Under Simulated And Actual Stress) database. This database includes:

  • Simulated subcorpus: Speech from 9 speakers with 11 speaking styles (including simulated emotions like angry, clear, fast, slow, soft, and conditions like cond50, cond70).
  • Actual subcorpus: Speech from 11 speakers under 5 real stress conditions (neutral, medst, hist, freefall, scream).

The SUSAS database was chosen for its large set of target emotions, availability of speaker IDs, and inclusion of both simulated and actual emotional states. For training and evaluation, the data was split into non-overlapping training, testing, and development sets, with different speakers used across these sets for speaker-independent experiments.

Key Insights and Ideas

  • Modeling the Full PDF Matters: The key insight is that modeling the entire probability distribution of prosodic features (via GMM supervectors) is more effective than relying on a fixed set of summary statistics (as in the baseline). This allows the system to capture more nuanced prosodic variations related to emotion.
  • UBM for Speaker Normalization: Using a Universal Background Model (UBM) for MAP adaptation of GMMs is a powerful technique for learning emotion-independent prosodic distributions, which helps in reducing the impact of speaker variability.
  • Fusion Boosts Performance: Combining the proposed GMM-SVM system with the baseline system yields significant performance gains, indicating that the two methods capture complementary information about emotions in speech.
  • Speaker Dependence is Key: Speaker-dependent models generally achieve better performance than speaker-independent models, highlighting the persistent challenge of speaker variability in emotion recognition, even with advanced techniques.

Key Results

  • In a speaker-dependent framework, the proposed prosodic GMM-SVM approach achieved a relative improvement of over 14% in Equal Error Rate (EER) compared to the baseline on the SUSAS Simulated subcorpus.
  • When combining the proposed system and the baseline, a relative improvement of over 17% was observed compared to the baseline alone.
  • For speaker-independent experiments, the proposed system also showed improvements, with the combined approach outperforming the baseline.
  • The paper notes that the Actual subcorpus generally yielded better results than the Simulated subcorpus, likely due to a smaller number of target classes making discrimination easier.
  • Speaker-dependent experiments consistently showed better performance, with the combined system achieving relative improvements of around 30.64% (Actual) and 43.21% (Simulated) compared to the baseline.

Limitations and Future Work

  • Speaker Inter-Variability: While the UBM helps, speaker inter-variability remains a significant challenge, as evidenced by the performance gap between speaker-dependent and independent setups.
  • Emotion Set Definition: The performance is shown to be dependent on the specific emotion set used, with more distinguishable emotions yielding better results.
  • Future Work: The authors suggest exploring new configurations for continuous pitch estimation and, importantly, combining prosodic features with acoustic-level features for even more comprehensive emotion recognition.

Summary updated

BibTeX
@inproceedings{lopezmoreno2009speaker,
  author = {Ignacio Lopez-Moreno and Carlos Ortego-Resa and Joaquin Gonzalez-Rodriguez and Daniel Ramos},
  title = {{Speaker dependent emotion recognition using prosodic supervectors}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure