Gender and affect recognition based on GMM and GMM-UBM modeling with relevance MAP estimation

Rok Gajšek, Janez Žibert, Tadej Justin, Vitomir Štruc, Boštjan Vesnicer, France MiheličInterspeech 2010

TL;DR

This paper presents a system for recognizing gender and affect (emotion) from speech, leveraging Gaussian Mixture Models (GMMs) and a Universal Background Model (UBM) with Maximum A Posteriori (MAP) adaptation. The approach shows promise, particularly for gender recognition, and highlights the potential of frame-level acoustic features for paralinguistic tasks.

What Problem Does This Paper Address?

Modern human-computer interaction (HCI) systems aim to be more intuitive and human-like. A key aspect of human interaction is the ability to infer information beyond the literal words spoken, such as the speaker's gender and emotional state. This paper tackles the challenge of automatically recognizing these paralinguistic cues from speech. Specifically, it addresses the INTERSPEECH 2010 Paralinguistic Challenge, which included sub-challenges for gender and affect recognition. The authors aim to improve the naturalness of HCI systems by incorporating this inferred information.

Methods

The core of the proposed system relies on Gaussian Mixture Models (GMMs), a standard probabilistic model for representing data distributions.

For Gender Recognition, the approach is straightforward:

  • Separate GMMs are trained for each gender category (male, female, child) using acoustic features.
  • The system classifies a new speech sample by determining which GMM best matches its acoustic characteristics.

For Affect (Emotion) Recognition, a more sophisticated GMM-based approach is employed:

  • A Universal Background Model (UBM) is first trained on all available training data. This UBM acts as a general model of speech acoustics.
  • For each individual speech sample, the UBM is adapted using Maximum A Posteriori (MAP) estimation. This process refines the UBM's parameters (specifically, the means of the Gaussian components) to better represent the specific acoustic characteristics of that sample. This adapted model effectively creates a "super vector" of means, which serves as the feature vector for the sample.
  • The acoustic features used are low-level descriptors (LLDs) extracted at the frame level. These LLDs are split into four subsets: prosodic features, Mel-Frequency Cepstral Coefficients (MFCCs), Linear Spectral Frequency Pairs (LSPs), and Log Mel Frequency Bands (LMFBs).
  • Each of these four feature subsets is modeled independently using the GMM-UBM MAP approach.
  • Finally, the predictions from these four sub-systems are fused using a sum rule fusion to produce the final affect recognition output.
  • The authors also explored Support Vector Regression (SVR) as an alternative to a baseline regression method for affect recognition, finding comparable performance.

Datasets

The paper utilizes data from the INTERSPEECH 2010 Paralinguistic Challenge. This includes:

  • The "aGender" corpus for the Gender Sub-Challenge.
  • A dataset for the Affect Sub-Challenge, which was split into training, development, and test sets.

The authors note that the development set was used for tuning parameters and evaluating performance, while the test set was used for final, "blinded" evaluation.

Key Insights and Ideas

  • Frame-level features are effective for paralinguistic tasks: The success in both gender and affect recognition suggests that analyzing acoustic features on a frame-by-frame basis, rather than relying solely on aggregated functionals, can capture crucial paralinguistic information.
  • GMM-UBM with MAP adaptation shows promise for affect recognition: This technique, borrowed from speaker recognition, proves to be a viable approach for modeling the nuances of emotional speech.
  • Fusion of multiple feature types improves performance: Combining predictions from different acoustic feature subsets (prosodic, MFCC, LSP, LMFB) via sum rule fusion led to better results in the affect recognition task.
  • Gender recognition is more robust than affect recognition: The results indicate a clearer distinction between gender categories compared to the subtle variations in emotional states.
  • Potential for overfitting to development data: The authors acknowledge that parameters tuned on the development set might not generalize perfectly to the test set, as evidenced by performance drops.

Key Results

  • Gender Sub-Challenge:
    • On the test set, the system achieved an unweighted average recall of 82.84%.
  • The confusion matrix for the development set showed high accuracy for male speech (96.18%), good accuracy for female speech (88.24%), but significant confusion with children's speech (56.53%).
  • The final system's results on the test set were slightly better than the baseline system.
  • Affect Sub-Challenge:
    • On the test set, the system achieved a cross-correlation (CC) of 0.390 and a mean linear error (MLE) of 0.143.
  • While the system's performance on the development set was superior to the baseline, the test set performance showed a drop, with the baseline system achieving a higher CC (0.421 vs. 0.390).
  • The MLE for the system on the test set was slightly better than the baseline (0.143 vs. 0.146).

Limitations and Future Work

  • Generalization to unseen data: The authors observed a performance drop on the test set compared to the development set, suggesting that the system might be overfitted to the development data or that there's a discrepancy between the test and development/training sets.
  • Robustness of the affect recognition system: The system's performance on the test set was lower than the baseline, indicating a potential lack of robustness.
  • Future Work:
    • Increasing training data: The authors suggest that incorporating more training data, potentially from other databases, could help in estimating a more robust UBM and mitigating overfitting.
  • Further investigation into feature fusion: Exploring different fusion strategies or more advanced modeling techniques for combining sub-systems could yield better results.
  • Addressing the gender/child speech confusion: Further research is needed to improve the distinction between female and children's speech.

Summary updated

BibTeX
@inproceedings{gajsek2010gender,
  author = {Rok Gajšek and Janez Žibert and Tadej Justin and Vitomir Štruc and Boštjan Vesnicer and France Mihelič},
  title = {{Gender and affect recognition based on GMM and GMM-UBM modeling with relevance MAP estimation}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure