A Temporal-Spectral-Based Squeeze-and- Excitation Feature Fusion Network for Motor Imagery EEG Decoding

Yang Li, Lianghui Guo, Yu Liu, Jingyu Liu, Fangang MengIEEE Transactions on Neural Systems and Rehabilitation Engineering 2021 · DOI

Abstract

Motor imagery (MI) electroencephalography (EEG) decoding plays an important role in brain-computer interface (BCI), which enables motor-disabled patients to communicate with the outside world via external devices. Recent deep learning methods, which fail to fully explore both deep-temporal characterizations in EEGs itself and multi-spectral information in different rhythms, generally ignore the temporal or spectral dependencies in MI-EEG. Also, the lack of effective feature fusion probably leads to redundant or irrelative information and thus fails to achieve the most discriminative features, resulting in the limited MI-EEG decoding performance. To address these issues, in this paper, a MI-EEG decoding framework is proposed, which uses a novel temporal-spectral-based squeeze-and-excitation feature fusion network (TS-SEFFNet). First, the deep-temporal convolution block (DT-Conv block) implements convolutions in a cascade architecture, which extracts high-dimension temporal representations from raw EEG signals. Second, the multi-spectral convolution block (MS-Conv block) is then conducted in parallel using multi-level wavelet convolutions to capture discriminative spectral features from corresponding clinical subbands. Finally, the proposed squeeze-and-excitation feature fusion block (SE-Feature-Fusion block) maps the deep-temporal and multi-spectral features into comprehensive fused feature maps, which highlights channel-wise feature responses by constructing interdependencies among different domain features. Competitive experimental results on two public datasets demonstrate that our method is able to achieve promising decoding performance compared with the state-of-the-art methods.

Introduction

Brain-Computer Interfaces (BCIs) allow individuals with motor disabilities to communicate or control external devices by imagining movements, a process known as Motor Imagery (MI). However, decoding these electroencephalography (EEG) signals is notoriously difficult due to a low signal-to-noise ratio, non-stationarity, and significant inter-subject variability.

Existing deep learning models often struggle with two primary issues: incomplete feature extraction and poor feature fusion. Many models focus on either temporal patterns or spectral rhythms in isolation, failing to integrate them deeply. Furthermore, when models do combine different data types, they often rely on naive fusion methods like simple concatenation, which can introduce redundant or irrelevant information that degrades performance.

Proposed Architecture: TS-SEFFNet

The authors propose the Temporal-Spectral-Based Squeeze-and-Excitation Feature Fusion Network (TS-SEFFNet) to address these challenges. The architecture is built around three specialized blocks designed to extract and intelligently combine multi-domain features.

Overview of the TS-SEFFNet architecture, highlighting the parallel DT-Conv and MS-Conv blocks followed by the SE-Feature-Fusion block.

Deep-Temporal Convolution (DT-Conv) Block

This branch utilizes a cascade of temporal convolution units to extract high-dimensional, long-range temporal representations directly from raw EEG signals. This deep architecture allows the model to capture complex temporal dependencies that shallower models might miss.

Multi-Spectral Convolution (MS-Conv) Block

Operating in parallel to the temporal branch, this block uses Wavelet Convolutions (WaveConv). By employing Daubechies 4 (Db4) wavelets, the model extracts features corresponding to five clinical frequency bands ($\delta, \theta, \alpha, \beta, \gamma$). This approach bakes neuroscientific domain knowledge into the network, ensuring it focuses on frequencies relevant to motor imagery without adding unnecessary learnable parameters.

SE-Feature-Fusion Block

Instead of simple concatenation, this block employs a Squeeze-and-Excitation (SE) mechanism. This attention-based unit calculates the interdependencies between different feature channels and assigns higher weights to the most discriminative ones. By prioritizing the most informative features from both the temporal and spectral branches, the SE block effectively filters out noise and redundancy.

Experimental Results

The model was evaluated on two widely recognized public datasets: the BCI Competition 2008 IV 2a (4-class MI) and the High Gamma Dataset (HGD).

TS-SEFFNet outperformed several state-of-the-art (SOTA) baseline models, including Shallow ConvNet and Deep ConvNet. On the BCI IV 2a dataset, it reached an average accuracy of 74.71%, while on the High Gamma Dataset, it achieved a significant 93.25% accuracy. The model also demonstrated superior stability, achieving the highest Cohen’s kappa coefficients across both datasets (0.663 for BCI IV 2a and 0.910 for HGD).

Ablation study showing the impact of the number of temporal convolution units and kernel size on decoding accuracy for both the BCI IV 2a (a) and HGD (b) datasets.

Beyond accuracy, the model is computationally efficient with approximately 282k parameters. Inference time for a single trial is roughly 27–36 milliseconds, making it well-suited for real-time BCI applications where low latency is critical.

Conclusion and Future Work

TS-SEFFNet demonstrates that combining deep temporal features with multi-spectral wavelet features through an attention-based fusion mechanism significantly improves MI-EEG decoding. Future work will focus on developing adaptive channel-selection methods to better capture spatial dependencies and exploring transfer learning techniques to improve cross-subject decoding performance, reducing the need for extensive subject-specific training.

Summary updated

Abstract

Motor imagery (MI) electroencephalography (EEG) decoding plays an important role in brain-computer interface (BCI), which enables motor-disabled patients to communicate with the outside world via external devices. Recent deep learning methods, which fail to fully explore both deep-temporal characterizations in EEGs itself and multi-spectral information in different rhythms, generally ignore the temporal or spectral dependencies in MI-EEG. Also, the lack of effective feature fusion probably leads to redundant or irrelative information and thus fails to achieve the most discriminative features, resulting in the limited MI-EEG decoding performance. To address these issues, in this paper, a MI-EEG decoding framework is proposed, which uses a novel temporal-spectral-based squeeze-and-excitation feature fusion network (TS-SEFFNet). First, the deep-temporal convolution block (DT-Conv block) implements convolutions in a cascade architecture, which extracts high-dimension temporal representations from raw EEG signals. Second, the multi-spectral convolution block (MS-Conv block) is then conducted in parallel using multi-level wavelet convolutions to capture discriminative spectral features from corresponding clinical subbands. Finally, the proposed squeeze-and-excitation feature fusion block (SE-Feature-Fusion block) maps the deep-temporal and multi-spectral features into comprehensive fused feature maps, which highlights channel-wise feature responses by constructing interdependencies among different domain features. Competitive experimental results on two public datasets demonstrate that our method is able to achieve promising decoding performance compared with the state-of-the-art methods.
BibTeX
@article{li2021temporal,
  author = {Yang Li and Lianghui Guo and Yu Liu and Jingyu Liu and Fangang Meng},
  title = {{A Temporal-Spectral-Based Squeeze-and- Excitation Feature Fusion Network for Motor Imagery EEG Decoding}},
  journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  year = {2021},
  doi = {10.1109/tnsre.2021.3099908},
}
Figure