Multimodal Fusion of EEG and Musical Features in Music-Emotion Recognition

Nattapong Thammasan, Ken-ichi Fukui, Masayuki NumaoAAAI 2017 · DOI

Abstract

This study investigates a multimodal approach to music-emotion recognition by integrating Electroencephalogram (EEG) signals with intrinsic musical features. Utilizing decision-level fusion, the researchers demonstrate that combining internal physiological responses with external musical stimuli significantly enhances the accuracy of predicting emotional states, particularly arousal and valence. The results highlight the "anchor" effect of musical features in mitigating the instability of EEG data, especially in subject-independent classification scenarios.

Introduction

Recognizing human emotions through technology remains a significant challenge due to the subjective nature of emotional experience and the complexity of physiological signals. While Electroencephalogram (EEG) sensors provide a direct window into brain activity, the signals are inherently non-stable, varying significantly over time and across individuals. Conversely, music contains structural cues—such as tempo, key, and dynamics—that suggest specific emotional states, though individual interpretation remains subjective. This paper addresses the challenge of multimodal fusion: combining internal physiological responses (EEG) with external stimuli (musical features) to create a robust model for tracking emotional arousal (intensity) and valence (pleasantness).

Methodology

The researchers developed a processing pipeline for two distinct data streams:

  1. EEG Processing: Data was collected from 12 electrodes focused on the frontal lobe. The complexity of these brainwaves was quantified using the Fractal Dimension (FD) approach via the Higuchi algorithm. Additionally, "asymmetry indexes" were calculated to compare activity between the left and right hemispheres, a standard proxy for emotional processing.
  2. Musical Feature Extraction: Using the MIRtoolbox, 37 features were extracted from MIDI songs, categorized into dynamics (RMS amplitude), rhythm (tempo, onset slopes), timbre (spectral roughness, MFCCs), and tonal features (key clarity, harmonic changes).
  3. Decision-Level Fusion: Separate Support Vector Machine (SVM) classifiers were trained for EEG and musical data. Their outputs were combined using a weighted linear formula: $$P_{multimodal} = \alpha P_{EEG} + (1 - \alpha) P_{music}$$ By varying the weighting factor $\alpha$ from 0 to 1, the study evaluated the optimal balance between physiological and stimulus-based data.

Results

Performance was measured using the Matthews Correlation Coefficient (MCC) to account for data class imbalances. The study found that multimodal fusion consistently outperformed single-modality models.

MCC performance across arousal and valence for subject-dependent and subject-independent models as a function of the weighting factor α.

Key findings include:

  • The "Anchor" Effect of Music: Musical features often outperformed EEG, particularly in subject-independent scenarios. Music provides a stable baseline that compensates for the inherent noise in cross-subject brainwave patterns.
  • Arousal vs. Valence Sensitivity: In subject-independent arousal classification, performance dropped sharply as the weight of EEG ($\alpha$) increased. This suggests that for arousal, external musical features like tempo and volume are more reliable predictors than raw EEG across different individuals.
  • Subject-Dependent Success: Personalized models achieved significantly higher accuracy, with MCC scores exceeding 0.7 for arousal, proving that individual-specific tuning is superior to general models.

Discussion and Conclusion

The results suggest that EEG and musical features play complementary roles, representing "felt" versus "expressed" emotion, respectively. While the linear fusion model proved effective, future work may explore more sophisticated deep learning-based integration to capture complex non-linear relationships.

Limitations of the current study include a small, non-diverse demographic (12 male participants) and the use of MIDI files rather than complex real-world audio. Expanding the cohort and utilizing high-fidelity audio recordings will be essential for generalizing these findings to broader applications in affective computing and music therapy.

Summary updated

BibTeX
@inproceedings{thammasan2017multimodal,
  author = {Nattapong Thammasan and Ken-ichi Fukui and Masayuki Numao},
  title = {{Multimodal Fusion of EEG and Musical Features in Music-Emotion Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.11112},
}
Figure