Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Debadatta Dash, Paul Ferrari, Jun WangFrontiers in Neuroscience 2020 · DOI

Abstract

Speech production is a hierarchical mechanism involving the synchronization of the brain and the oral articulators, where the intention of linguistic concepts is transformed into meaningful sounds. Individuals with locked-in syndrome (fully paralyzed but aware) lose their motor ability completely including articulation and even eyeball movement. The neural pathway may be the only option to resume a certain level of communication for these patients. Current brain-computer interfaces (BCIs) use patients' visual and attentional correlates to build communication, resulting in a slow communication rate (a few words per minute). Direct decoding of imagined speech from the neural signals (and then driving a speech synthesizer) has the potential for a higher communication rate. In this study, we investigated the decoding of five imagined and spoken phrases from single-trial, non-invasive magnetoencephalography (MEG) signals collected from eight adult subjects. Two machine learning algorithms were used: an artificial neural network (ANN) with statistical features as the baseline and convolutional neural networks (CNNs) applied on the spatial, spectral, and temporal features extracted from the MEG signals. Experimental results indicated the possibility to decode imagined and spoken phrases directly from neuromagnetic signals. CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.

Introduction

For individuals suffering from "locked-in syndrome," cognitive awareness remains intact while motor control is entirely lost. Communication for these patients is currently limited to slow, attention-based Brain-Computer Interfaces (BCIs) that often rely on eye-tracking or selecting letters on a screen. The "holy grail" of BCI research is the direct decoding of speech intentions from neural activity. While invasive methods like Electrocorticography (ECoG) offer high signal quality, they require risky surgical implantation.

This study explores Magnetoencephalography (MEG) as a powerful non-invasive alternative. MEG measures the magnetic fields produced by neuronal activity, offering superior spatial and temporal resolution compared to Electroencephalography (EEG) because magnetic fields are not distorted by the skull. By leveraging deep learning, this research aims to decode complex phrases directly from MEG signals.

Experimental Design and Methodology

The study involved eight healthy, right-handed adult participants. Brain activity was recorded using a 306-channel Elekta Neuromag TRIUX system.

The experimental setup featuring a subject in the 306-channel MEG scanner with a visual stimulus screen.

Participants were tasked with processing five common English phrases used in assistive communication: "Do you understand me," "That’s perfect," "How are you," "Good-bye," and "I need help." Each trial followed a specific temporal structure to isolate different stages of speech processing.

The experimental protocol timeline, illustrating the sequence of rest, perception, preparation (imagination), and production (articulation).

As shown in the protocol above, each trial consisted of:

  1. Rest (0.5s): Pre-stimulus baseline.
  2. Perception (1s): Reading the target phrase on a screen.
  3. Preparation/Imagination (1s): Mentally rehearsing the phrase without moving articulators.
  4. Production/Articulation (2s): Speaking the phrase aloud.

Neural Decoding Architecture

The technical innovation of this research lies in treating neural signals as multi-dimensional "images." The raw data underwent wavelet denoising (Daubechies db-4) to isolate specific neural oscillations.

To capture the complex relationships in the data, the researchers developed a Spatial-Spectral-Temporal CNN (SST-CNN) approach. They generated scalograms—visual representations of signal energy across time and frequency—using Continuous Wavelet Transform (CWT). By arranging scalograms from 196 sensors into a 14x14 matrix, a single trial was transformed into a multi-channel image. This allowed the use of state-of-the-art computer vision models, including AlexNet, ResNet101, and Inception-ResNet-v2, to classify the linguistic intentions.

Results and Key Insights

The SST-CNN approach significantly outperformed the baseline Artificial Neural Network (ANN) across all experimental stages.

Stage ANN Accuracy SST-CNN (AlexNet) Accuracy
Perception 71.95% 90.38%
Imagination 80.83% 93.24%
Production 90.55% 96.65%

A critical finding was the high accuracy during the "imagination" phase (93.24%). This suggests a neural "pre-play" effect, where the brain's preparation for speech contains nearly as much decodable information as the physical act of speaking. AlexNet emerged as the most efficient model, likely because the energy patterns in scalograms are better suited to its architecture than the deeper, more complex ResNet models designed for natural images.

Limitations and Future Directions

While these results demonstrate the feasibility of high-accuracy MEG decoding, several challenges remain for clinical application:

  • Vocabulary Expansion: The current system is limited to five phrases; moving toward an open vocabulary is essential for natural communication.
  • Generalizability: Models were subject-specific. Future work must address cross-subject calibration to create "plug-and-play" systems.
  • Hardware Portability: Traditional MEG requires large, cooled installations. The authors highlight Optically Pumped Magnetometers (OPMs)—wearable, room-temperature MEG sensors—as the likely platform for future mobile BCIs.

Ultimately, this study proves that non-invasive neuromagnetic signals contain rich information for decoding both imagined and overt speech, paving the way for faster, more natural communication for paralyzed individuals.

Summary updated

Abstract

Speech production is a hierarchical mechanism involving the synchronization of the brain and the oral articulators, where the intention of linguistic concepts is transformed into meaningful sounds. Individuals with locked-in syndrome (fully paralyzed but aware) lose their motor ability completely including articulation and even eyeball movement. The neural pathway may be the only option to resume a certain level of communication for these patients. Current brain-computer interfaces (BCIs) use patients' visual and attentional correlates to build communication, resulting in a slow communication rate (a few words per minute). Direct decoding of imagined speech from the neural signals (and then driving a speech synthesizer) has the potential for a higher communication rate. In this study, we investigated the decoding of five imagined and spoken phrases from single-trial, non-invasive magnetoencephalography (MEG) signals collected from eight adult subjects. Two machine learning algorithms were used. One was an artificial neural network (ANN) with statistical features as the baseline approach. The other was convolutional neural networks (CNNs) applied on the spatial, spectral and temporal features extracted from the MEG signals. Experimental results indicated the possibility to decode imagined and spoken phrases directly from neuromagnetic signals. CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.
BibTeX
@inproceedings{dash2020decoding,
  author = {Debadatta Dash and Paul Ferrari and Jun Wang},
  title = {{Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals}},
  booktitle = {Frontiers in Neuroscience},
  year = {2020},
  doi = {10.3389/fnins.2020.00290},
}
Figure