Using i-Vector Space Model for Emotion Recognition

Rui Xia, Yang LiuInterspeech 2012 · OpenReview

Abstract

Speech emotion recognition (SER) is often hindered by the high dimensionality of acoustic features and the limited availability of emotional training data. This paper adapts the i-vector framework, widely successful in speaker and language identification, to the task of SER. The authors propose an "extended i-vector" approach that leverages multiple emotion-specific total variability spaces. Experimental results on the USC AudioVisual and IEMOCAP datasets show that the proposed method outperforms traditional GMM supervectors and high-dimensional static feature sets, providing a more compact and discriminative representation of emotional speech.

Introduction

Automatically identifying human emotion from speech is a complex challenge with applications in human-computer interaction and mental health monitoring. Historically, two main approaches have dominated the field:

  1. Massive Static Feature Sets: These involve extracting thousands of statistical functionals (e.g., mean, variance, quartiles) from low-level descriptors like pitch and energy. While effective, these high-dimensional vectors are often redundant and computationally expensive.
  2. GMM Supervectors: These are formed by concatenating the parameters of Gaussian Mixture Models (GMMs). However, their extreme dimensionality makes them difficult to train effectively with the relatively small datasets typical of emotional speech research.

This paper explores whether the i-vector model—which compresses high-dimensional GMM supervectors into a low-dimensional "total variability space"—can be repurposed to capture the acoustic nuances of emotion.

Methodology: The Extended i-Vector Approach

The core innovation of this work is the Extended i-vector framework. Unlike traditional speaker recognition systems that use a single universal background model (UBM), this approach utilizes emotion-specific modeling to enhance discriminative power.

Flowchart of the proposed extended i-vector extraction process

The process involves several key steps:

  1. Universal Background Model (UBM): A GMM is first trained on a large corpus of neutral speech (e.g., Switchboard) to represent general speech characteristics.
  2. Emotion-Specific GMMs: The neutral UBM is adapted using Maximum A Posteriori (MAP) estimation toward specific emotion categories (e.g., Angry, Happy, Sad).
  3. Total Variability Training: For each emotion category, a "Total Variability Matrix" ($T$) is learned. This matrix defines a low-dimensional space capturing the primary variations within that emotional class.
  4. Extraction & Concatenation: For a given utterance, an i-vector is extracted for each emotion model. These individual vectors are then concatenated into a single "extended i-vector."
  5. Classification: The final compact vector is classified using a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel.

Experimental Setup

The researchers evaluated the system on two distinct datasets:

  • Acted Speech: The USC AudioVisual dataset, containing 123 sentences read by an actress across four emotional states.
  • Spontaneous Speech: The IEMOCAP database, featuring 941 utterances from actors in scripted and improvised interactions.
  • Background Data: The Switchboard corpus provided the data for training the initial neutral UBM.

Results and Discussion

The proposed extended i-vector method consistently outperformed baseline systems across both acted and spontaneous speech categories.

Performance comparison of the proposed i-vector system against baselines

Key Insights

  • Efficiency and Dimensionality: The extended i-vector (typically 160 dimensions) is nearly ten times smaller than the standard openSMILE static feature set (1584 dimensions), yet it achieves higher accuracy. This indicates that i-vectors are more efficient at encoding emotional signals while filtering out acoustic noise.
  • Complementary Information: Combining the i-vector system with traditional static features yielded the highest overall performance (92.0% for acted, 74.6% for spontaneous). This suggests that i-vectors capture acoustic information that is distinct from standard statistical functionals.
  • Discriminative Power: By measuring the "distance" of an utterance from multiple emotional profiles rather than just a neutral baseline, the system gains a more nuanced understanding of emotional variance.

Conclusion and Future Work

The adaptation of the i-vector framework to emotion recognition represents a significant step forward in creating robust, low-dimensional feature representations. Future work will focus on applying noise-reduction techniques common in speaker ID, such as Within-Class Covariance Normalization (WCCN) and Linear Discriminant Analysis (LDA), and testing the model on larger, more diverse "in-the-wild" datasets.

Summary updated

BibTeX
@inproceedings{xia2012using,
  author = {Rui Xia and Yang Liu},
  title = {{Using i-Vector Space Model for Emotion Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=Lcj_EpVSWhM},
}
Figure