Online Learning for Wearable EEG-Based Emotion Classification

Sidratul Moontaha, Franziska Elisabeth Friederike Schumann, Bert Arnrich

TL;DR

This paper introduces a real-time, online learning pipeline for classifying emotions (valence and arousal) using wearable EEG devices. It demonstrates that this approach can achieve high accuracy even with delayed labels, paving the way for practical applications like seizure prediction and improved human-computer interaction.

What Problem Does This Paper Address?

Accurately understanding and interpreting human emotions is crucial for advancing human-computer interaction (HCI), enabling early detection of mental health issues, and even predicting events like epileptic seizures. While Electroencephalography (EEG) is a powerful tool for capturing brain activity related to emotions, existing emotion classification methods often rely on offline, computationally intensive techniques that require complete datasets and are unsuitable for real-time applications. Furthermore, many studies use specialized, expensive EEG equipment, limiting their applicability to everyday scenarios. This paper addresses the need for a robust, real-time emotion classification system that can work with affordable, wearable EEG devices and adapt to continuously streaming data, even when ground truth labels are not immediately available.

Methods

The core of this work is an online learning pipeline designed for real-time emotion classification. Key methodological aspects include:

  • Wearable EEG Integration: The pipeline is designed to work with consumer-grade EEG devices like the Muse S headband and Neurosity Crown, which are more accessible and portable than traditional lab equipment.
  • Online Learning: Instead of training on a static dataset, the pipeline continuously updates its models as new data arrives. This is crucial for adapting to the non-stationary nature of EEG signals and individual user variations.
  • Progressive Validation (Test-Then-Train): This evaluation strategy is employed to ensure that models are trained and tested on unseen data, mimicking real-world scenarios where data arrives sequentially.
  • Delayed Label Handling: A significant contribution is the pipeline's ability to handle delayed labels, which is common in real-world applications where the ground truth (e.g., a participant's self-reported emotion) might only become available after a period of time (e.g., after watching a video).
  • Feature Extraction: Power Spectral Density (PSD) features from different frequency bands, along with power ratios, are extracted from the EEG data within tumbling windows.
  • Machine Learning Classifiers: The pipeline evaluates three online learning algorithms: Adaptive Random Forest (ARF), Streaming Random Patches (SRP), and Logistic Regression (LR). ARF and SRP, being ensemble methods with drift detection capabilities, generally outperformed LR.

Datasets

The study utilizes the following datasets:

  • AMIGOS Dataset: A publicly available, curated EEG dataset for affect, personality, and mood research. This dataset was used for initial validation of the pipeline with immediate labels.
  • Newly Curated Datasets (Dataset II & III): The authors collected new data using two consumer-grade EEG devices (Muse S and Neurosity Crown) from 15 participants who watched emotional videos. This new data was used to validate the pipeline in more realistic, live settings, including scenarios with delayed labels.

Key Insights and Ideas

  • Real-time Emotion is Achievable with Wearables: The study demonstrates that high-performance emotion classification is feasible using affordable, wearable EEG devices, moving beyond lab-based setups.
  • Online Learning is Essential for Dynamic Data: The non-stationary nature of EEG signals necessitates online learning approaches that can adapt to changing data patterns and individual users over time.
  • Delayed Labels are a Practical Hurdle: Real-world applications often involve delayed labels. The pipeline's ability to handle this gracefully, while still achieving reasonable performance, is a critical step towards practical deployment.
  • Ensemble Methods Excel in Online Settings: ARF and SRP, as ensemble classifiers, showed superior performance in the online learning context compared to Logistic Regression.
  • Window Size Matters: A smaller window size (1 second) generally yielded better classification performance, suggesting a trade-off between capturing short-term emotional fluctuations and the amount of data needed for robust feature extraction.

Key Results

  • AMIGOS Dataset Performance: The proposed pipeline achieved mean F1-Scores of over 80% for both valence and arousal on the AMIGOS dataset, outperforming previous baseline results.
  • New Dataset Performance (Immediate Labels): On the newly curated Dataset II, the pipeline achieved mean F1-Scores of over 82% with ARF and SRP classifiers, validating its effectiveness with consumer-grade devices.
  • Live Classification Performance (Delayed Labels): In live classification scenarios with delayed labels (Dataset III), the highest F1-Scores reached 73% for arousal and 60% for valence. While lower than immediate label settings, these results are considered promising for real-world applications.
  • Device Comparison: While there were slight differences in performance between the Muse S and Neurosity Crown devices, these were not statistically significant, suggesting the pipeline is robust across different consumer EEG hardware.

Limitations and Future Work

  • Performance with Delayed Labels: The performance significantly drops when labels are delayed, indicating a need for further optimization in handling such scenarios.
  • Class Imbalance: The dataset exhibited a strong class imbalance, with high arousal ratings being significantly more prevalent. This can impact model performance and suggests the need for techniques like up-sampling.
  • Stimulus Duration: The length of video stimuli contributed to label delays. Future work could involve shortening stimuli or using more dynamic pre-processing and feature selection.
  • Multi-modal Integration: The authors suggest exploring additional modalities like heart rate and electrodermal activity to potentially improve prediction accuracy.
  • Participant Feedback: Making predictions visible to participants could enable neuro-feedback applications.
  • Basic Emotions: Classifying basic emotions instead of valence/arousal dimensions might be a more tractable problem.

Summary updated

Abstract

Emotions are indicators of affective states and play a significant role in human daily life, behavior, and interactions. Giving emotional intelligence to the machines could, for instance, facilitate early detection and prediction of (mental) diseases and symptoms. Electroencephalography (EEG) -based emotion recognition is being widely applied because it measures electrical correlates directly from the brain rather than the indirect measurement of other physiological responses initiated by the brain. The recent development of non-invasive and portable EEG sensors makes it possible to use them in real-time applications. Therefore, this paper presents a real-time emotion classification pipeline, which trains different binary classifiers for the dimensions of Valence and Arousal from an incoming EEG data stream. After achieving a 23.9% (Arousal) and 25.8% (Valence) higher f1-score on the state-of-art AMIGOS dataset, this pipeline was applied to the dataset achieved by an emotion elicitation experimental framework developed within the scope of this thesis. Following two different protocols, 15 participants were recorded using two different consumer-grade EEG devices while watching 16 short emotional videos in a controlled environment. For an immediate label setting, the mean f1-score of 87% and 82% were achieved for Arousal and Valence, respectively. In a live scenario, while continuously being updated on the incoming data stream with delayed labels, the pipeline proved to be fast enough to achieve predictions in real time. However, the significant discrepancy from the readily available labels on the classification scores leads to future work to include more data with frequent delayed labels in the live settings.
BibTeX
@misc{moontaha2023online,
  author = {Sidratul Moontaha and Franziska Elisabeth Friederike Schumann and Bert Arnrich},
  title = {{Online Learning for Wearable EEG-Based Emotion Classification}},
  year = {2023},
  doi = {10.3390/s23052387},
}
Figure