Speech decoding using cortical and subcortical electrophysiological signals

Hemmings Wu, Chengwei Cai, Wenjie Ming, Wangyu Chen, Zhoule Zhu, Feng Chen, Hongjie Jiang, Zhe Zheng, Mohamad Sawan, Ting Wang, Junming ZhuFrontiers in Neuroscience 2024 · DOI

Abstract

This study underscores the essential roles of both cortical and subcortical structures in different aspects of speech decoding.

Introduction

Most current speech-decoding brain-computer interfaces (BCIs) focus on the cerebral cortex, often utilizing Electrocorticography (ECoG). However, speech production involves a multi-layered neural network including deep-seated subcortical structures that coordinate fine motor movements. This research addresses the "blind spot" of subcortical contributions, specifically in the context of Mandarin Chinese. As a tonal language, Mandarin requires precise decoding of pitch contours (tones) in addition to articulatory features, presenting a unique challenge for neural decoding.

Experimental Design and Methodology

The study involved two native Mandarin speakers undergoing surgery for refractory epilepsy. Researchers utilized stereotactic electroencephalography (sEEG) to record signals from both cortical and subcortical regions, including the thalamus, hippocampus, and amygdala.

Schematic diagram of the experimental design to record vocal and electrophysiological signals simultaneously.

Participants performed a reading task involving 407 Chinese characters, covering a comprehensive range of Mandarin pronunciations and tones. Neural features were extracted from three frequency bands, with a focus on the High Gamma band (70–150 Hz). A 3-layer bidirectional Long Short-Term Memory (bLSTM) recurrent neural network was employed to predict articulatory place, articulatory manner, and linguistic tone.

Neural Mapping and Electrode Placement

The use of sEEG allowed for precise localization of activity across 290 electrode contacts. This spatial resolution was critical for identifying the functional specialization of different brain regions.

Reconstructed CT images showing the sEEG implants in the two patients, highlighting the coverage of both cortical and subcortical areas.

Key Insights and Results

The study revealed distinct roles for cortical and subcortical structures:

  • Functional Specialization: The Superior Temporal Gyrus (STG) was most effective for decoding articulatory place (86.5% accuracy), while the Thalamus proved superior for decoding linguistic tone (58.3% accuracy).
  • Subcortical Contribution: The thalamus performed comparably to the STG in decoding articulatory manner (~51.7% accuracy), demonstrating that subcortical signals provide robust information for speech decoding.
  • Synergistic Effects: Combining signals from both cortical (STG) and subcortical (Thalamus) regions resulted in the highest overall decoding accuracy, particularly for tone. This suggests that these regions offer complementary rather than redundant information.
  • Frequency Bands: High Gamma activity (70–150 Hz) was the most predictive feature across all regions, confirming its importance in both cortical and subcortical speech processing.

Conclusion and Future Directions

This research provides a proof-of-concept for integrating subcortical signals into speech BCIs. While limited by a small sample size and clinical constraints (such as right-hemisphere electrode placement), the findings suggest that "listening" to the brain's depths can significantly enhance the decoding of complex linguistic features like tone. Future work will focus on larger cohorts, diverse languages, and the long-term stability of these deep-brain signals for clinical BCI applications.

Summary updated

Abstract

This study underscores the essential roles of both cortical and subcortical structures in different aspects of speech decoding.
BibTeX
@inproceedings{wu2024speech,
  author = {Hemmings Wu and Chengwei Cai and Wenjie Ming and Wangyu Chen and Zhoule Zhu and Feng Chen and Hongjie Jiang and Zhe Zheng and Mohamad Sawan and Ting Wang and Junming Zhu},
  title = {{Speech decoding using cortical and subcortical electrophysiological signals}},
  booktitle = {Frontiers in Neuroscience},
  year = {2024},
  doi = {10.3389/fnins.2024.1345308},
}
Figure