A neurophysiologically interpretable deep neural network predicts complex movement components from brain activity

Neelesh Kumar, Konstantinos P. MichmizosScientific 2022 · DOI

TL;DR

This paper introduces a novel 3D Convolutional Neural Network (3D-CNN) that decodes complex hand movement components (reaction time, active/passive mode, direction) from EEG data with high accuracy. Crucially, the network is designed to be neurophysiologically interpretable, aligning its predictions with known brain activity patterns, which enhances reliability for real-world applications like neurorehabilitation.

What Problem Does This Paper Address?

Decoding human movement from non-invasive electroencephalography (EEG) is vital for applications like neurorehabilitation robots and neural prosthetics. While deep learning (DL) excels at real-time data decoding, its application to EEG is hampered by several issues. Existing datasets often contain only gross motor tasks, easily solvable by simpler networks. Furthermore, current DL models lack a strong connection to underlying neurophysiology, limiting their generalizability and trustworthiness. This "black box" nature makes it difficult to understand why a network makes a certain prediction, hindering its deployment in critical applications. The authors aim to bridge this gap by developing a DL model that not only decodes complex movement components accurately but also provides interpretable insights aligned with neuroscience.

Methods

The core innovation is a 3-dimensional Convolutional Neural Network (3D-CNN). Unlike traditional 2D-CNNs that treat EEG as a flattened matrix, this 3D-CNN processes topography-preserving EEG inputs. This means the network explicitly considers the spatial relationships between EEG sensors, treating the data as a 3D volume (channels x time x features).

Key methodological aspects include:

  • 3D Convolutional Layers: These layers extract spatiotemporal features by convolving across both sensor space and time.
  • Topography-Preserving Input Representation: EEG sensor data is mapped onto a 2D spatial matrix, preserving neighborhood relationships between sensors. Peripheral channels are dropped to ensure consistent row sizes.
  • Gradient-weighted Class Activation Maps (Grad-CAM): This technique is used to visualize which EEG sensors and time segments were most influential in the network's classification decisions, providing neurophysiological interpretability.
  • Leave-One-Subject-Out (LOSO) Cross-Validation: This rigorous evaluation method assesses the network's generalization ability to unseen subjects, crucial for real-world BCI applications.

Datasets

The study utilized a newly acquired dataset from an in-house motor experiment involving 12 healthy subjects.

  • Experiment: Subjects performed goal-directed arm movements on a plane using a back-drivable rehabilitation robot.
  • Movement Components Classified:
    • Reaction Time (RT): Discretized into "slow" and "fast" classes.
  • Mode of Movement: Classified as "active" (subject-driven) or "passive" (robot-assisted).
  • Movement Direction: Classified into four orthogonal directions: "left," "right," "up," and "down."
  • Dataset Composition:
    • RT dataset: ~50 trials per class per subject.
  • Active/Passive dataset: ~208 trials per class per subject.
  • Directions dataset: ~52 trials per direction per subject.

Key Insights and Ideas

  • Spatial-Temporal Dependencies are Crucial: The 3D-CNN's ability to capture spatiotemporal dependencies across brain areas, rather than just temporal patterns, is key to decoding complex movements. This highlights that the where and when of brain activity are both critical.
  • Neurophysiological Interpretability Enhances Reliability: By aligning network decisions with known brain regions involved in motor planning and execution (e.g., M1, PMC, SMA), the Grad-CAM analysis provides confidence in the model's predictions. This is a significant step beyond "black box" models.
  • Pre-Movement Activity is Highly Discriminative: The study found significant class-discriminatory differences in EEG signals before the actual movement onset, particularly related to preparatory slow cortical potentials (SCP) and movement-related cortical potentials (MRCP). This suggests that the brain's preparation phase holds rich information for decoding intent.
  • 3D Representation Outperforms 2D: The explicit 3D input representation and architecture led to better classification accuracies compared to a modern 2D-CNN, demonstrating the advantage of preserving spatial relationships.

Key Results

  • High Classification Accuracies: The 3D-CNN achieved impressive average leave-one-subject-out (LOSO) test accuracies:
  • Reaction Time (RT): 79.81%
    • Active vs. Passive: 81.23%
    • Directions: 82.00%
  • Superior Performance: The proposed 3D-CNN outperformed a modern 2D-CNN by 1.1% to 6.74% across the classification tasks. It also significantly outperformed LSTMs.
  • Interpretable Insights: Grad-CAM analysis successfully identified brain regions and time segments crucial for classification, aligning with established neurophysiological knowledge of motor planning and execution. For instance, it highlighted the role of the supplementary motor area (SMA), premotor cortex (PMC), and primary motor cortex (M1).
  • Robustness: The network showed low variability in performance across subjects despite high inter-subject variability in EEG data, suggesting good generalization. Subject-specific training further boosted accuracies to over 90% for some tasks.

Limitations and Future Work

  • Dataset Size: While a new dataset was introduced, the authors note that the number of trials per subject was limited for some classifications, impacting subject-specific training.
  • Generalizability to Impaired Populations: The current study was conducted on healthy subjects. The authors acknowledge that intrinsic variability in neurological disorders might hinder generalization, suggesting the need for domain adaptation techniques.
  • Real-time Inference Speed: While the model achieved good accuracy, further optimization for real-time applications is implied, especially concerning inference time.
  • Future Directions:
    • Applying domain adaptation techniques to improve generalization to impaired populations.
  • Further exploring the interpretability of the network to gain deeper insights into brain function.
  • Extending the approach to decode other complex brain activities in real-time.
  • Investigating online artifact removal and motion detection for seamless real-time prediction.

Summary updated

Abstract

The effective decoding of movement from non-invasive electroencephalography (EEG) is essential for informing several therapeutic interventions, from neurorehabilitation robots to neural prosthetics. Deep neural networks are most suitable for decoding real-time data but their use in EEG is hindered by the gross classes of motor tasks in the currently available datasets, which are solvable even with network architectures that do not require specialized design considerations. Moreover, the weak association with the underlying neurophysiology limits the generalizability of modern networks for EEG inference. Here, we present a neurophysiologically interpretable 3-dimensional convolutional neural network (3D-CNN) that captured the spatiotemporal dependencies in brain areas that get co-activated during movement. The 3D-CNN received topography-preserving EEG inputs, and predicted complex components of hand movements performed on a plane using a back-drivable rehabilitation robot, namely (a) the reaction time (RT) for responding to stimulus (slow or fast), (b) the mode of movement (active or passive, depending on whether there was an assistive force provided by the apparatus), and (c) the orthogonal directions of the movement (left, right, up, or down). We validated the 3D-CNN on a new dataset that we acquired from an in-house motor experiment, where it achieved average leave-one-subject-out test accuracies of 79.81%, 81.23%, and 82.00% for RT, active vs. passive, and direction classifications, respectively. Our proposed method outperformed the modern 2D-CNN architecture by a range of 1.1% to 6.74% depending on the classification task. Further, we identified the EEG sensors and time segments crucial to the classification decisions of the network, which aligned well with the current neurophysiological knowledge on brain activity in motor planning and execution tasks. Our results demonstrate the importance of biological relevance in networks for an accurate decoding of EEG, suggesting that the real-time classification of other complex brain activities may now be within our reach.
BibTeX
@inproceedings{kumar2022neurophysiologically,
  author = {Neelesh Kumar and Konstantinos P. Michmizos},
  title = {{A neurophysiologically interpretable deep neural network predicts complex movement components from brain activity}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-05079-0},
}
Figure