The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition

Enshi Zhang, Rafael Trujillo, Christian PoellabauerACL 2024 · DOI

Abstract

Speech Emotion Recognition (SER) is vital for advanced human-computer interaction, yet it is often hindered by the lack of naturalistic data and effective multimodal fusion techniques. This paper introduces MERSA (Multimodal Emotion Recognition and Sentiment Analysis), a dataset featuring 150 participants in "in the wild" settings. We also propose a transformer-based architecture that fuses audio (wav2vec 2.0) and text (BERT) features through a series of LSTM layers. Our results demonstrate that this approach achieves state-of-the-art competitive performance in predicting continuous emotional dimensions—valence, arousal, and dominance—across MERSA and standard benchmarks like IEMOCAP and MSP-PODCAST.

Introduction

The field of Speech Emotion Recognition (SER) faces two primary challenges: the "naturalness gap" and the difficulty of multimodal integration. Most existing datasets rely on lab-simulated or scripted performances (e.g., movies), which fail to capture the subtle nuances of daily human emotion. Furthermore, while combining audio and text modalities is known to improve accuracy, effectively fusing these high-dimensional features while preserving temporal context remains a significant research hurdle.

The MERSA Dataset

To address the data scarcity, we introduce the MERSA dataset. Collected from 150 participants over a two-week period, it comprises:

  • 37.42 hours of naturalistic audio recordings.
  • 2,650 self-reported surveys based on the PANAS (Positive and Negative Affect Schedule) scale.
  • Nearly 50,000 hours of physiological data, including heart rate and sleep patterns from wearable devices.

A key contribution is the mapping of PANAS scores to the 3D dimensional emotional space (Valence, Arousal, Dominance), providing an objective ground truth for continuous emotion prediction.

Proposed Methodology

Our model architecture leverages pre-trained transformers for feature extraction and Recurrent Neural Networks (RNNs) for temporal modeling.

Proposed transformer-based fusion architecture with LSTM layers.

Feature Extraction and Fusion

We utilize wav2vec 2.0 for acoustic features and BERT for linguistic features. These representations are mean-pooled and concatenated into a 1536-dimensional vector (early fusion).

Temporal Modeling with LSTMs

The fused features are processed through five LSTM layers (512 units each). This configuration is designed to capture the temporal "flow" of emotion, which pure transformer models often struggle to represent as effectively. The final output provides continuous values for Valence, Arousal, and Dominance.

Experimental Results

The proposed wav2vec2-base + BERT-base + LSTM configuration was evaluated across three datasets. A significant finding was that the inclusion of LSTM layers consistently outperformed transformer-only baselines, underscoring the importance of recurrent structures in emotional progression modeling.

Performance comparison (CCC) across IEMOCAP, MSP-PODCAST, and MERSA datasets.

As shown in the results, the model maintains high Concordance Correlation Coefficient (CCC) scores even in cross-domain scenarios. For instance, when trained on MSP-PODCAST and tested on MERSA, it achieved a CCC of 0.641 for Valence and 0.593 for Arousal, indicating robust generalization to real-world data. Additionally, upgrading to the "Large" version of wav2vec 2.0 improved arousal prediction by approximately 7.75%, highlighting the value of high-resolution acoustic modeling for emotional intensity.

Conclusion and Future Work

This work demonstrates the efficacy of combining transformer-based feature extraction with LSTM-based temporal modeling for naturalistic SER. While the MERSA dataset provides a wealth of physiological data, the current model focuses on audio and text. Future research will explore the integration of "body signals" (e.g., heart rate) and aim to increase demographic diversity beyond the current college-student-centric sample. Optimization for real-time deployment on edge devices also remains a priority.

Summary updated

BibTeX
@inproceedings{zhang2024mersa,
  author = {Enshi Zhang and Rafael Trujillo and Christian Poellabauer},
  title = {{The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.752},
}
Figure