Emotion Recognition Using Multimodal Deep Learning

Wei Liu, Wei-Long Zheng, Bao-Liang Lu

Abstract

Emotion recognition is a critical component of advanced Human-Machine Interfaces (HMI). Traditional methods often struggle with the inherent noise and lack of robustness in single physiological modalities. This paper discusses a novel approach using multimodal deep learning, specifically Bimodal Deep AutoEncoders (BDAE), to fuse electroencephalogram (EEG) signals and eye-tracking data. By extracting a shared representation across these diverse signals, the proposed model achieves state-of-the-art performance, reaching 91.01% accuracy on the SEED dataset.

Introduction

The ability of machines to accurately perceive human emotions is essential for natural interaction. However, physiological signals like EEG and eye movements are complex and subject to individual variability. While multimodal approaches aim to increase reliability, traditional "shallow" fusion techniques—such as simple feature concatenation or decision-level averaging—often fail to model the non-linear correlations between different biological responses. This research explores whether deep learning can identify a "common language" or shared representation between brain activity and ocular behavior to enhance emotion recognition.

Methodology: Deep Multimodal Fusion

The core of the proposed framework relies on two deep architectures based on Restricted Boltzmann Machines (RBMs):

  1. Bimodal Deep AutoEncoder (BDAE): This architecture features two input branches for EEG and eye-tracking data. These branches converge into a central hidden layer that learns a "shared representation." This layer captures the underlying emotional state reflected in both modalities simultaneously.
  2. Deep AutoEncoder (DAE) for Enhancement: This model is designed for scenarios where only one modality is available. By training the network to reconstruct both modalities from a single input, the DAE uses the learned correlations to "enhance" or clean the available features, effectively leveraging cross-modal knowledge.
Architectures for (a) unimodal enhancement and (b) bimodal fusion using shared representations.

The training involves unsupervised pre-training followed by supervised fine-tuning. A linear Support Vector Machine (SVM) is then applied to the learned shared representations for final classification.

Experimental Results

The models were evaluated on two benchmark datasets: SEED (62-channel EEG and eye-tracking) and DEAP (32-channel EEG and peripheral signals).

Performance Benchmarks

The multimodal approach demonstrated significant improvements over existing state-of-the-art (SOTA) methods:

  • SEED Dataset: The BDAE achieved 91.01% accuracy, surpassing the previous SOTA of 87.59%.
  • DEAP Dataset: The model reached 83.25% accuracy, outperforming benchmarks such as Rozgic et al. (76.9%).
  • Unimodal Enhancement: Utilizing a DAE improved standalone EEG accuracy from 77.54% to 81.19%, proving that the network successfully transfers knowledge from the missing modality.

Insights into Shared Representations

A key finding was the effectiveness of cross-modal learning. In experiments where a classifier was trained on shared representations from EEG and tested on those from eye-tracking, accuracy reached ~66% (well above the 33% chance level). This indicates that the BDAE learns a universal "emotion feature" that transcends specific biological sensors. Furthermore, multimodal fusion reduced the standard deviation of results, indicating a more stable and reliable system compared to unimodal counterparts.

Discussion and Future Work

While the results prove the efficacy of deep multimodal fusion, challenges remain. The "black box" nature of shared representations limits interpretability. Future research will focus on dissecting the relationship between raw features and deep representations to understand the specific physiological markers the model prioritizes. Additionally, the sensitivity of the BDAE to hyperparameters and its long-term stability across different environments require further investigation.

Conclusion

This work demonstrates that multimodal deep learning can effectively bridge the gap between disparate physiological signals. By learning shared representations, BDAEs provide a robust and highly accurate solution for emotion recognition, paving the way for more empathetic and responsive human-machine systems.

Summary updated

Abstract

To enhance the performance of affective models and reduce the cost of acquiring physiological signals for real-world applications, we adopt multimodal deep learning approach to construct affective models with SEED and DEAP datasets to recognize different kinds of emotions. We demonstrate that high level representation features extracted by the Bimodal Deep AutoEncoder (BDAE) are effective for emotion recognition. With the BDAE network, we achieve mean accuracies of 91.01 % and 83.25 % on SEED and DEAP datasets, respectively, which are much superior to those of the state-of-the-art approaches. By analysing the confusing matrices, we found that EEG and eye features contain complementary information and the BDAE network could fully take advantage of this complement property to enhance emotion recognition.
BibTeX
@inproceedings{liu2016emotion,
  author = {Wei Liu and Wei-Long Zheng and Bao-Liang Lu},
  title = {{Emotion Recognition Using Multimodal Deep Learning}},
  booktitle = {ICONIP},
  year = {2016},
  doi = {10.1007/978-3-319-46672-9\_58},
}

Loading related content...

Figure