Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription

Saurabh Sahu, Vikramjit Mitra, Nadee Seneviratne, Carol Espy-WilsonInterspeech 2019

TL;DR

This paper explores how combining audio and text information from speech can improve emotion recognition, particularly for the "valence" (pleasantness) dimension. They demonstrate that using Automatic Speech Recognition (ASR) outputs instead of perfect ground truth transcriptions introduces a performance drop, but the multi-modal approach still outperforms audio-only systems, highlighting the robustness of this method.

What Problem Does This Paper Address?

Recognizing emotions from speech is a long-standing challenge in human-computer interaction. While acoustic features (pitch, loudness, etc.) are good at capturing arousal (intensity), they struggle with valence (pleasantness). Textual information, derived from speech transcriptions, has shown promise for sentiment analysis and thus could help improve valence prediction. However, previous work often relied on expensive, manually created "ground truth" transcriptions. This paper aims to bridge this gap by investigating how multi-modal learning (audio + text) performs when using ASR-generated transcriptions, which are more practical for real-world applications, and to analyze the specific contributions of audio and text modalities to emotion recognition.

Methods

The authors employ a multi-modal deep learning architecture. For the audio modality, they extract features using two sets: eGeMAPS (prosodic and spectral features) and features from pyAudioAnalysis (claimed to be more speaker-agnostic). These features are fed into a sequence of Long Short-Term Memory (LSTM) layers. For the text modality, they use pre-trained GloVe embeddings to represent words, which are then fed into a similar LSTM architecture. The outputs from the audio and text LSTMs are concatenated and passed through a dense layer for final classification.

A key aspect of their methodology is the comparison of using perfect "ground truth" transcriptions versus transcriptions generated by two different ASR APIs (Google and Wit.ai). They also conduct a cross-corpus study to assess the generalizability of their trained models.

Datasets

  • IEMOCAP (Interactive Emotional Dyadic Motion Capture): This dataset, comprising approximately 7 hours of audio, was used for training and cross-validation. It includes categorical emotion labels (happy, sad, angry, neutral) and continuous ratings for arousal and valence. The authors used a 5-fold leave-one-session-out cross-validation strategy.
  • MSP-IMPROV: This dataset was used for a cross-corpus study, serving as the test set while IEMOCAP was used for training. It contains similar emotion categories but has an unbalanced class distribution.

Key Insights and Ideas

  • Text for Valence, Audio for Arousal: The paper strongly supports the hypothesis that audio features are more effective for predicting arousal (e.g., distinguishing "anger" which is high arousal), while text features are better for predicting valence (e.g., distinguishing "happy" which is high valence from "sad" which is low valence).
  • Multi-modal Synergy: Combining audio and text modalities consistently improves class-wise accuracies compared to unimodal systems, suggesting that the two modalities offer complementary information.
  • ASR Transcription Trade-offs: While using ASR transcriptions instead of ground truth leads to a performance drop, the multi-modal system still significantly outperforms audio-only systems. This indicates that even imperfect transcriptions can provide valuable information for emotion recognition.
  • ASR Model Generalizability: The study suggests that modern, deep learning-based ASR models, trained on vast amounts of data, exhibit good generalizability, producing useful transcriptions for unseen datasets. This is crucial for building practical, end-to-end emotion recognition systems.
  • Impact of Transcription Quality: The quality of ASR transcriptions directly impacts the performance of the multi-modal system. Lower quality transcriptions lead to more confusion between classes with similar arousal values, implying that audio features play a more dominant role in disambiguation in such cases.

Key Results

  • Unimodal Performance: Text-based models achieved higher accuracies for "happy" (high valence) and "neutral" emotions, while audio-based models performed better on "anger" (high arousal).
  • Multi-modal Improvement: The multi-modal system using ground truth transcriptions showed improved class-wise accuracies across the board compared to unimodal systems.
  • ASR vs. Ground Truth: Using Google's ASR transcriptions resulted in a relative accuracy loss of 4% compared to ground truth, while Wit.ai's transcriptions led to a 5.3% loss.
  • Cross-Corpus Performance: When trained on IEMOCAP and tested on MSP-IMPROV, the multi-modal system with ASR transcriptions showed a relative improvement of 9.8% (Google) and 5.2% (Wit.ai) over an audio-only model, though the gains were less pronounced than in cross-validation due to domain differences.

Limitations and Future Work

  • ASR API Limitations: The ASR APIs did not transcribe all utterances, leading to different subsets of data being used for each API. This could affect direct comparisons.
  • Domain Differences: The cross-corpus study highlighted that vocabulary differences between datasets can impact performance, suggesting a need for more robust cross-domain adaptation.
  • Future Directions: The authors propose exploring articulatory features, investigating alternative word embeddings (e.g., sub-word embeddings), developing emotion-specific word embeddings, and exploring text features derived from emotion-specific dictionaries. They also aim to explore novel ways to combine audio and text information within the multi-modal framework.

Summary updated

BibTeX
@inproceedings{sahu2019multi,
  author = {Saurabh Sahu and Vikramjit Mitra and Nadee Seneviratne and Carol Espy-Wilson},
  title = {{Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure