Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotions

Vasudha Kowtha, Vikramjit Mitra, Chris Bartels, Erik Marchi, Sue Booker, William Caruso, Sachin Kajarekar, Devang Naik

Introduction

While modern voice assistants excel at speech-to-text transcription, they often fail to capture the emotional nuance of vocal expression. Distinguishing an urgent request from a casual observation requires more than linguistic understanding; it requires paralinguistic analysis. Traditional approaches rely on "categorical" labels like "happy" or "angry," which are difficult for humans to annotate consistently. This research shifts the focus toward "emotion primitives"—Valence, Arousal, and Dominance (VAD)—which offer a more consistent framework for machine learning models to interpret human emotion.

Methodology

The researchers evaluated two primary neural network architectures for processing sequential acoustic data. The first is a standard Long Short-Term Memory (LSTM) network. The second, more sophisticated model is the Time Convolutional LSTM (TC-LSTM). The TC-LSTM incorporates a convolutional layer before the LSTM to capture local temporal patterns and includes a "skip connection" that feeds raw acoustic features directly into the recurrent layer.

Comparison of (a) standard LSTM and (b) the proposed TC-LSTM architecture with a convolutional layer and skip connection.

Models were trained using various acoustic features, including Mel-filterbank energies (MFB), Pitch (F0), and Articulatory features (TV). The optimization process utilized the Concordance Correlation Coefficient (CCC) as a loss function to maximize the agreement between predicted scores and human ground-truth annotations.

Datasets and Generalization

The study leveraged two major datasets: an internal 120-hour US English dataset and the MSP-Podcast (ver-3) dataset, which contains 50 hours of spontaneous, non-acted speech. A critical finding was the impact of training diversity on model performance. Models trained on a single dataset often struggled with the "valence gap"—the inherent difficulty in predicting how positive or negative a sound is from audio alone.

Visualization of predicted valence for models trained on a single dataset (left) versus multiple datasets (right), showing improved separation and gradient consistency.

Key Results and Insights

  • Architectural Superiority: The TC-LSTM consistently outperformed the standard LSTM, demonstrating that capturing local temporal context via convolutions is highly beneficial for emotion recognition tasks.
  • Generalization Gains: Training on a combination of datasets (Internal + MSP) improved the CCC for valence detection by 30% compared to models trained on a single source.
  • The Arousal Indicator: When attempting to discern categorical emotions (e.g., anger or surprise) from predicted primitives, arousal proved to be the most reliable indicator across all categories.
  • Feature Efficiency: Low-dimensional, frame-level features like simple filterbank energies performed nearly as well as high-dimensional, complex feature sets, suggesting high efficiency for real-world applications.

Conclusion and Future Work

Despite significant progress, a "domain mismatch" remains; models often see a 40% performance drop when tested on unfamiliar datasets. While human-labeled valence is excellent for identifying categorical emotions, predicted valence from current models still lags behind. Future research will focus on closing this gap to bring speech-based emotion detection closer to the level of human perception.

Summary updated

BibTeX
@inproceedings{kowtha2020detecting,
  author = {Vasudha Kowtha and Vikramjit Mitra and Chris Bartels and Erik Marchi and Sue Booker and William Caruso and Sachin Kajarekar and Devang Naik},
  title = {{Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053079},
}
Figure