Acoustic emotion recognition using dynamic Bayesian networks and multi-space distributions

R. Barra-Chicote, Fernando Fernández, S. Lutfi, Juan Manuel Lucas-Cuesta, J. Macias-Guarasa, J. M. Montero, R. San-Segundo, J. M. PardoInterspeech 2009

TL;DR

This paper presents a novel system for recognizing emotions in speech by leveraging Dynamic Bayesian Networks (DBNs) and a Multi-Space Distribution (MSD) framework. The system achieves statistically significant improvements in emotion recognition accuracy, particularly for spontaneous, real-world speech, by effectively modeling the temporal dynamics and the dual nature of acoustic features.

What Problem Does This Paper Address?

Recognizing emotions in speech is a challenging task, especially when dealing with spontaneous, non-acted speech recorded in realistic environments. Unlike acted emotions, spontaneous emotions are more nuanced and influenced by factors like cultural background. Existing research often relies on acted speech, which may not generalize well to real-world scenarios. Furthermore, acoustic features like fundamental frequency (F0) have a dual nature, being present in voiced speech but absent in unvoiced segments. Effectively modeling this duality and the temporal evolution of emotional cues is crucial for accurate emotion recognition. This paper addresses these challenges by proposing a system designed for realistic emotional speech, incorporating temporal modeling and a method to handle the voiced/unvoiced nature of F0.

Methods

The core of the proposed system is a Dynamic Bayesian Network (DBN), a probabilistic graphical model well-suited for capturing temporal dependencies in sequential data. The DBN is used to integrate multiple streams of acoustic information.

Key methodological innovations include:

  • Multi-Space Distribution (MSD) for F0: To handle the voiced/unvoiced nature of the fundamental frequency (F0), the authors employ a Multi-Space Distribution approach. This allows the model to explicitly account for the presence or absence of F0, treating voiced and unvoiced segments differently. This is a novel way to model prosodic information that is crucial for emotion recognition.
  • Feature Set: The system utilizes a combination of segmental and prosodic features. Specifically, it includes:
  • MFCCs (Mel-Frequency Cepstral Coefficients): The first 12 MFCCs and their first and second derivatives (representing spectral characteristics of the vocal tract).
  • Energy: Logarithm of energy and its derivatives (representing loudness).
  • F0 (Fundamental Frequency): Logarithm of F0 and its first and second derivatives (representing pitch).
  • F0 Normalization: To mitigate speaker-specific biases in F0, a per-speaker normalization is applied during training, and a global normalization is used for testing.
  • DBN Architecture: The DBN models the temporal evolution of these features. Discrete parent variables are used to switch between different models for voiced and unvoiced segments, influencing how the continuous acoustic features are processed.

Datasets

The primary dataset used is the FAU Aibo Emotion Corpus. This corpus is notable for containing spontaneous, emotionally colored speech from children interacting with an electronic pet in realistic environments. It comprises nine hours of speech from 51 children, recorded in two different schools. The corpus provides detailed transcriptions, including word boundaries, non-linguistic vocalizations, and emotion labels. The authors used partitions of this corpus for training, development, and testing, as described in Tables 1 and 2.

Key Insights and Ideas

  • MSD is Crucial for F0: The MSD approach for modeling F0 significantly improves performance, demonstrating its importance in capturing the nuances of prosodic information, especially in spontaneous speech. The authors show a relative improvement of over 10.2% for the 2-class problem when using MSD.
  • DBNs Excel at Temporal Modeling: Dynamic Bayesian Networks provide a robust framework for integrating diverse acoustic features and modeling their temporal evolution, which is essential for understanding the dynamic nature of emotional expression.
  • Real-World Speech is Harder: The results highlight the inherent difficulty of recognizing emotions in spontaneous, unacted speech compared to acted speech, as evidenced by the performance metrics.
  • Feature Selection Matters: The system achieves competitive results with a relatively small set of features (45 elements), suggesting that careful feature selection and modeling are more critical than sheer feature quantity.

Key Results

  • 2-Class Task: The system achieved an unweighted recall of 67.06% on the internal test set. This performance is comparable to the challenge baseline and shows a relative improvement of 1.5% over the baseline using dynamic modeling.
  • 5-Class Task: The system achieved an unweighted recall of 38.24% on the internal test set. This represents a statistically significant relative improvement of 6.5% over the challenge baseline that also used dynamic modeling.
  • MSD Validation: The MSD approach showed a relative improvement of 10.2% for the 2-class problem compared to a baseline without MSD.
  • Static vs. Dynamic Modeling: When compared to a static modeling approach, the proposed DBN-based system showed minimal differences, but importantly, it achieved this with a significantly smaller feature set.

Limitations and Future Work

  • Limited Linguistic Information: The system relies solely on acoustic features due to the unavailability of accurate German acoustic and language models. The authors acknowledge that integrating linguistic and supralinguistic knowledge would likely lead to further improvements.
  • Speaker Normalization: While speaker bias in F0 is addressed, further exploitation of multi-stream capabilities and detailed studies on speaker-related dependencies and normalization strategies are suggested for future work.
  • Future Integration: The proposed DBN architecture is flexible and can readily incorporate additional information sources like speech recognition outputs or speaker identification, which are planned for future research.

Summary updated

BibTeX
@inproceedings{barrachicote2009acoustic,
  author = {R. Barra-Chicote and Fernando Fernández and S. Lutfi and Juan Manuel Lucas-Cuesta and J. Macias-Guarasa and J. M. Montero and R. San-Segundo and J. M. Pardo},
  title = {{Acoustic emotion recognition using dynamic Bayesian networks and multi-space distributions}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure