A Deep Neural Network for SSVEP-Based Brain-Computer Interfaces

Osman Berke Güney, Muhtasham Oblokulov, Huseyin Ozkan

TL;DR

This paper introduces a novel deep neural network (DNN) architecture for SSVEP-based Brain-Computer Interfaces (BCIs) that achieves state-of-the-art performance in target identification for BCI spellers. By processing raw EEG signals end-to-end and employing a unique two-stage training approach, the proposed DNN significantly outperforms existing methods, enabling faster and more accurate communication for individuals with severe motor disabilities.

What Problem Does This Paper Address?

Brain-Computer Interfaces (BCIs) offer a lifeline for individuals with severe motor impairments, enabling them to communicate and control external devices using their brain signals. A key application is the BCI speller, which allows users to type by selecting characters on a screen. One popular BCI paradigm, Steady-State Visual Evoked Potentials (SSVEP), relies on presenting characters that flicker at distinct frequencies. The brain's response to these flickering stimuli, the SSVEP signal, contains spectral components at the stimulus frequency and its harmonics. The challenge in SSVEP spellers is to accurately and rapidly identify which character the user intends to select, which translates to maximizing the Information Transfer Rate (ITR). Existing methods often struggle with inter-subject variability, require extensive preprocessing, or are limited in their ability to capture complex signal dynamics. This paper aims to address these limitations by proposing a more robust and efficient deep learning approach for SSVEP-based target identification.

Methods

The core of this work is a novel deep neural network (DNN) architecture designed to process multi-channel SSVEP signals directly in the time domain, from raw EEG input to final character prediction. The DNN comprises four convolutional layers and one fully connected layer, with each layer serving specific functions:

  • Layer 1 (Harmonics/Sub-bands Combination): This layer addresses the challenge of varying harmonic contributions to the SSVEP signal. Instead of manual selection or rule-based combination, the DNN learns to combine sub-bands of harmonics in a data-driven manner, allowing it to adapt to individual subject characteristics and optimize the use of different harmonic degrees.
  • Layer 2 (Channel Combination): Recognizing that different brain regions might be more responsive to specific SSVEP components, this layer learns to combine information from multiple EEG channels. It goes beyond simple linear combinations by allowing for a large number of combinations, which are then processed non-linearly in subsequent layers to avoid redundancy and capture complex spatial patterns.
  • Layer 3 (Filtering, Downsampling, Nonlinearity): This layer performs downsampling (halving the temporal resolution) and applies a non-linear activation (ReLU). This helps reduce computational complexity while introducing non-linearity crucial for processing the learned channel combinations effectively.
  • Layer 4 (Feature Extraction): This convolutional layer further processes the data to extract relevant features, likely tuned to the spectral patterns of different stimulus frequencies.
  • Fully Connected Layer: This final layer performs the multi-class classification to predict the target character.

A key innovation is the two-stage training approach:

  1. Global Model Training: The network is first trained on the combined data from all subjects. This allows it to learn general SSVEP characteristics and statistical commonalities across individuals.
  2. Subject-Specific Fine-tuning: The globally trained model is then used to initialize a new model, which is subsequently fine-tuned using data from each individual subject. This approach effectively leverages both generalizable patterns and individual specificities, leading to improved performance.

Datasets

The proposed DNN was evaluated on two publicly available, large-scale datasets:

  • Benchmark Dataset [14]: Consists of EEG data from 35 subjects with 40 target characters.
  • BETA Dataset [15]: Consists of EEG data from 70 healthy subjects with 40 target characters.

These datasets are crucial for validating the model's performance and generalizability across different subject populations and experimental setups.

Key Insights and Ideas

  • Data-Driven Harmonic and Channel Combination: The DNN learns optimal ways to combine harmonic sub-bands and EEG channels, moving away from manual or rule-based methods and allowing for greater adaptability and performance.
  • Two-Stage Training for Robustness: The combination of a global pre-training step followed by subject-specific fine-tuning is highly effective. It allows the model to learn generalizable SSVEP features while also adapting to individual brain signal variations, leading to significant performance gains.
  • End-to-End Processing: The network directly processes raw EEG signals, eliminating the need for extensive manual preprocessing steps, which simplifies the BCI pipeline and potentially captures more nuanced information.
  • Importance of Nonlinearity: The use of ReLU activations throughout the network is crucial for effectively processing the learned channel combinations and capturing complex interactions within the SSVEP signals.
  • Error Pattern Analysis: The study provides valuable insights into common error patterns, linking them to specific frequency relationships and the physical layout of the character matrix. This could inform future BCI speller design.

Key Results

The proposed DNN achieved impressive results on both datasets:

  • Benchmark Dataset: Achieved an Information Transfer Rate (ITR) of 265.23 bits/min with only 0.4 seconds of stimulation.
  • BETA Dataset: Achieved an ITR of 196.59 bits/min with only 0.4 seconds of stimulation.

These ITRs are reported as the highest ever achieved on these datasets, significantly outperforming state-of-the-art methods. The two-stage training approach improved performance by an average of 26% compared to using only the global or only the subject-specific model. The model also demonstrated efficient performance with a reduced number of channels (9 channels), offering a practical trade-off between accuracy and usability.

Limitations and Future Work

  • Calibration Time: While the fine-tuning stage is fast, a new user still requires an initial calibration period (estimated at 1-2 hours) to collect data for training. The authors suggest transferring a pre-trained global DNN model to mitigate the burden of the first stage training for new users.
  • Intermodulation (IM) Effects: The analysis of error patterns revealed that intermodulation effects, influenced by the character matrix layout, can significantly impact performance. Future work could explore optimizing the character matrix design to minimize these effects.
  • Generalizability: While the model shows strong performance on two large datasets, further validation on more diverse datasets and real-world BCI applications would be beneficial.
  • Real-time Implementation: The paper focuses on offline analysis. Adapting and optimizing the DNN for real-time BCI operation would be a crucial next step.

Summary updated

Abstract

Objective: Target identification in brain-computer interface (BCI) spellers refers to the electroencephalogram (EEG) classification for predicting the target character that the subject intends to spell. When the visual stimulus of each character is tagged with a distinct frequency, the EEG records steady-state visually evoked potentials (SSVEP) whose spectrum is dominated by the harmonics of the target frequency. In this setting, we address the target identification and propose a novel deep neural network (DNN) architecture. Method: The proposed DNN processes the multi-channel SSVEP with convolutions across the sub-bands of harmonics, channels, time, and classifies at the fully connected layer. We test with two publicly available large scale (the benchmark and BETA) datasets consisting of in total 105 subjects with 40 characters. Our first stage training learns a global model by exploiting the statistical commonalities among all subjects, and the second stage fine tunes to each subject separately by exploiting the individualities. Results: Our DNN achieves impressive information transfer rates (ITRs) on both datasets, 265.23 bits/min and 196.59 bits/min, respectively, with only 0.4 seconds of stimulation. The code is available for reproducibility at https://github.com/osmanberke/Deep-SSVEP-BCI. Conclusion: The presented DNN strongly outperforms the state-of-the-art techniques as our accuracy and ITR rates are the highest ever reported performance results on these datasets. Significance: Due to its unprecedentedly high speller ITRs and flawless applicability to general SSVEP systems, our technique has great potential in various biomedical engineering settings of BCIs such as communication, rehabilitation and control.
BibTeX
@article{guney2022deep,
  author = {Osman Berke Güney and Muhtasham Oblokulov and Huseyin Ozkan},
  title = {{A Deep Neural Network for SSVEP-Based Brain-Computer Interfaces}},
  journal = {IEEE Trans. Biomedical Engineering},
  year = {2022},
  doi = {10.1109/tbme.2021.3110440},
}
Figure