Abstract
Emotion recognition is a critical component of advanced Human-Machine Interfaces (HMI). Traditional methods often struggle with the inherent noise and lack of robustness in single physiological modalities. This paper discusses a novel approach using multimodal deep learning, specifically Bimodal Deep AutoEncoders (BDAE), to fuse electroencephalogram (EEG) signals and eye-tracking data. By extracting a shared representation across these diverse signals, the proposed model achieves state-of-the-art performance, reaching 91.01% accuracy on the SEED dataset.
Introduction
The ability of machines to accurately perceive human emotions is essential for natural interaction. However, physiological signals like EEG and eye movements are complex and subject to individual variability. While multimodal approaches aim to increase reliability, traditional "shallow" fusion techniques—such as simple feature concatenation or decision-level averaging—often fail to model the non-linear correlations between different biological responses. This research explores whether deep learning can identify a "common language" or shared representation between brain activity and ocular behavior to enhance emotion recognition.
Methodology: Deep Multimodal Fusion
The core of the proposed framework relies on two deep architectures based on Restricted Boltzmann Machines (RBMs):
- Bimodal Deep AutoEncoder (BDAE): This architecture features two input branches for EEG and eye-tracking data. These branches converge into a central hidden layer that learns a "shared representation." This layer captures the underlying emotional state reflected in both modalities simultaneously.
- Deep AutoEncoder (DAE) for Enhancement: This model is designed for scenarios where only one modality is available. By training the network to reconstruct both modalities from a single input, the DAE uses the learned correlations to "enhance" or clean the available features, effectively leveraging cross-modal knowledge.
The training involves unsupervised pre-training followed by supervised fine-tuning. A linear Support Vector Machine (SVM) is then applied to the learned shared representations for final classification.
Experimental Results
The models were evaluated on two benchmark datasets: SEED (62-channel EEG and eye-tracking) and DEAP (32-channel EEG and peripheral signals).
Performance Benchmarks
The multimodal approach demonstrated significant improvements over existing state-of-the-art (SOTA) methods:
- SEED Dataset: The BDAE achieved 91.01% accuracy, surpassing the previous SOTA of 87.59%.
- DEAP Dataset: The model reached 83.25% accuracy, outperforming benchmarks such as Rozgic et al. (76.9%).
- Unimodal Enhancement: Utilizing a DAE improved standalone EEG accuracy from 77.54% to 81.19%, proving that the network successfully transfers knowledge from the missing modality.
Insights into Shared Representations
A key finding was the effectiveness of cross-modal learning. In experiments where a classifier was trained on shared representations from EEG and tested on those from eye-tracking, accuracy reached ~66% (well above the 33% chance level). This indicates that the BDAE learns a universal "emotion feature" that transcends specific biological sensors. Furthermore, multimodal fusion reduced the standard deviation of results, indicating a more stable and reliable system compared to unimodal counterparts.
Discussion and Future Work
While the results prove the efficacy of deep multimodal fusion, challenges remain. The "black box" nature of shared representations limits interpretability. Future research will focus on dissecting the relationship between raw features and deep representations to understand the specific physiological markers the model prioritizes. Additionally, the sensitivity of the BDAE to hyperparameters and its long-term stability across different environments require further investigation.
Conclusion
This work demonstrates that multimodal deep learning can effectively bridge the gap between disparate physiological signals. By learning shared representations, BDAEs provide a robust and highly accurate solution for emotion recognition, paving the way for more empathetic and responsive human-machine systems.
Summary updated