Abstract
Speech Emotion Recognition (SER) is vital for advanced human-computer interaction, yet it is often hindered by the lack of naturalistic data and effective multimodal fusion techniques. This paper introduces MERSA (Multimodal Emotion Recognition and Sentiment Analysis), a dataset featuring 150 participants in "in the wild" settings. We also propose a transformer-based architecture that fuses audio (wav2vec 2.0) and text (BERT) features through a series of LSTM layers. Our results demonstrate that this approach achieves state-of-the-art competitive performance in predicting continuous emotional dimensions—valence, arousal, and dominance—across MERSA and standard benchmarks like IEMOCAP and MSP-PODCAST.
Introduction
The field of Speech Emotion Recognition (SER) faces two primary challenges: the "naturalness gap" and the difficulty of multimodal integration. Most existing datasets rely on lab-simulated or scripted performances (e.g., movies), which fail to capture the subtle nuances of daily human emotion. Furthermore, while combining audio and text modalities is known to improve accuracy, effectively fusing these high-dimensional features while preserving temporal context remains a significant research hurdle.
The MERSA Dataset
To address the data scarcity, we introduce the MERSA dataset. Collected from 150 participants over a two-week period, it comprises:
- 37.42 hours of naturalistic audio recordings.
- 2,650 self-reported surveys based on the PANAS (Positive and Negative Affect Schedule) scale.
- Nearly 50,000 hours of physiological data, including heart rate and sleep patterns from wearable devices.
A key contribution is the mapping of PANAS scores to the 3D dimensional emotional space (Valence, Arousal, Dominance), providing an objective ground truth for continuous emotion prediction.
Proposed Methodology
Our model architecture leverages pre-trained transformers for feature extraction and Recurrent Neural Networks (RNNs) for temporal modeling.
Feature Extraction and Fusion
We utilize wav2vec 2.0 for acoustic features and BERT for linguistic features. These representations are mean-pooled and concatenated into a 1536-dimensional vector (early fusion).
Temporal Modeling with LSTMs
The fused features are processed through five LSTM layers (512 units each). This configuration is designed to capture the temporal "flow" of emotion, which pure transformer models often struggle to represent as effectively. The final output provides continuous values for Valence, Arousal, and Dominance.
Experimental Results
The proposed wav2vec2-base + BERT-base + LSTM configuration was evaluated across three datasets. A significant finding was that the inclusion of LSTM layers consistently outperformed transformer-only baselines, underscoring the importance of recurrent structures in emotional progression modeling.
As shown in the results, the model maintains high Concordance Correlation Coefficient (CCC) scores even in cross-domain scenarios. For instance, when trained on MSP-PODCAST and tested on MERSA, it achieved a CCC of 0.641 for Valence and 0.593 for Arousal, indicating robust generalization to real-world data. Additionally, upgrading to the "Large" version of wav2vec 2.0 improved arousal prediction by approximately 7.75%, highlighting the value of high-resolution acoustic modeling for emotional intensity.
Conclusion and Future Work
This work demonstrates the efficacy of combining transformer-based feature extraction with LSTM-based temporal modeling for naturalistic SER. While the MERSA dataset provides a wealth of physiological data, the current model focuses on audio and text. Future research will explore the integration of "body signals" (e.g., heart rate) and aim to increase demographic diversity beyond the current college-student-centric sample. Optimization for real-time deployment on edge devices also remains a priority.
Summary updated