Abstract
Emotion recognition plays an important role in intelligent human-computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.
Introduction
Emotion recognition is a cornerstone of Affective Computing, yet it remains notoriously difficult for two primary reasons: low accuracy and subject dependence. Current systems often rely on a single modality, such as facial expressions, which can be easily masked or "faked" by a user. Conversely, physiological signals like electroencephalograms (EEG) offer a more objective "inside look" at emotional states but are highly variable between individuals—what signifies "joy" in one person's brainwaves might appear as "neutrality" in another's.
Furthermore, there is a lack of unified, open-source tools that allow researchers to easily collect data, train models, and visualize results in real-time across these disparate data streams. MindLink-Eumpy was developed to bridge this gap, providing a robust, real-time pipeline for multimodal emotion recognition.
System Architecture and Methodology
MindLink-Eumpy is designed as an end-to-end framework consisting of several integrated modules: Data Collection, Experimental Paradigm, and Algorithm Modules.
The technical core of the toolbox rests on three pillars:
- Facial Expression Detection: The toolbox utilizes a multitask Convolutional Neural Network (CNN). To address the challenge of small datasets, transfer learning is employed by pre-training the model on the large FER-2013 database before fine-tuning it on emotion-specific datasets.
- EEG Analysis: The framework offers two distinct paths for brain signal processing. The subject-dependent path uses a Support Vector Machine (SVM) tailored to an individual’s patterns. The subject-independent path utilizes a Long Short-Term Memory (LSTM) network designed to generalize across different users by capturing the temporal dynamics of EEG signals.
- Decision-Level Fusion: Rather than merging raw data (feature-level fusion), the system combines the predictions of the facial and EEG models. It uses techniques like Weight Enumerator (finding the optimal ratio of importance between face and brain data) and AdaBoost to produce a final, reliable emotional score.
Experimental Validation and Results
The researchers validated the toolbox using the DEAP and MAHNOB-HCI benchmark datasets, alongside an online experiment involving 15 healthy subjects. The study confirmed that the "whole is greater than the sum of its parts," as multimodal methods consistently outperformed single-modal ones.
In subject-dependent tests on the DEAP dataset, the system achieved 71.00% accuracy in valence and 72.14% in arousal. In subject-independent tests on MAHNOB-HCI, the LSTM-based method reached 78.56% accuracy in valence and 77.22% in arousal. During live online tests, the multimodal approach remained superior, with EEG-based SVMs proving particularly strong in detecting arousal levels compared to facial analysis alone.
Discussion and Conclusion
The success of MindLink-Eumpy highlights the synergy between "inside" physiological signals and "outside" expressions. EEG acts as a ground truth that fills gaps when facial expressions are ambiguous. While currently limited to EEG and facial images, future iterations of the toolbox may include eye-tracking, heart rate (ECG), or skin conductance to further boost accuracy.
As an open-source tool, MindLink-Eumpy is available for the community to extend, potentially serving as a standard benchmark for future human-computer interaction (HCI) research. By providing a unified platform for data collection, model training, and real-time visualization, it lowers the barrier to entry for sophisticated affective computing research.
Summary updated