TL;DR
This paper introduces the first publicly available bimodal dataset combining EEG and fMRI data for inner speech recognition. By merging the high temporal resolution of EEG with the high spatial resolution of fMRI, this dataset aims to advance brain-computer interfaces (BCIs) for individuals unable to speak, paving the way for improved speech prostheses.
What Problem Does This Paper Address?
The ability to decode "inner speech" – the silent, internal monologue we experience – holds immense potential for brain-computer interfaces (BCIs). Such BCIs could provide a voice for individuals who have lost the ability to speak due to conditions like ALS or stroke. However, a significant hurdle in developing these BCIs is the lack of comprehensive datasets that leverage the strengths of different neuroimaging techniques. Existing datasets are often unimodal, meaning they rely on either electroencephalography (EEG) for its excellent temporal resolution but poor spatial detail, or functional magnetic resonance imaging (fMRI) for its detailed spatial information but slow temporal response. This limitation restricts the potential accuracy and robustness of inner speech decoding. This paper addresses this gap by creating a novel bimodal dataset that combines EEG and fMRI, aiming to overcome the limitations of individual modalities and foster progress in inner speech recognition for BCI applications.
Methods
The core contribution of this paper is the creation and release of a unique bimodal EEG-fMRI dataset. The data were collected non-simultaneously from four healthy participants performing an inner-speech task. Participants were instructed to silently repeat presented words from either a social or numerical category.
- Experimental Design: The study involved an inner-speech task where participants silently vocalized words presented visually. This was performed across multiple sessions over three days.
- Data Acquisition:
- EEG: 64-channel EEG data was acquired using a BioSemi Active2 system at a sampling rate of 512 Hz.
- fMRI: Data was collected using a Siemens Magnetom Prisma MRI system with a 20-channel head coil. Visual stimuli were presented on a high-resolution display.
- Nonsimultaneous Acquisition: A key methodological choice was the non-simultaneous acquisition of EEG and fMRI data. This was done to avoid potential artifacts that can arise from simultaneous recording and to allow for optimization of the task protocol for each modality independently. A relaxation period of at least one hour was maintained between EEG and fMRI sessions.
- Preprocessing: Standard preprocessing steps were applied to both EEG and fMRI data. For EEG, this included artifact removal using Independent Component Analysis (ICA), which was particularly effective in reducing eye blink artifacts. For fMRI, preprocessing involved motion correction, slice-timing correction, coregistration, normalization to MNI space, and smoothing.
- Stimuli: Participants were presented with words from two categories: social (child, daughter, father, wife) and numerical (four, three, ten, six). Each of the 8 words was presented in 40 trials per modality, resulting in 320 trials per participant for both EEG and fMRI.
Datasets
This paper introduces a newly created bimodal EEG-fMRI dataset for inner-speech recognition.
- Dataset Name: Not explicitly named, but referred to as a "bimodal dataset containing EEG and fMRI data."
- Participants: Data was collected from four healthy, right-handed participants (two male, three female) aged 33-51. One participant's data (sub-04) was excluded due to high fluctuations during EEG recording.
- Modality Combination: The dataset uniquely combines EEG and fMRI data.
- Task: Inner-speech production.
- Stimuli: 8 words (4 social, 4 numerical).
- Trials: 320 trials per participant per modality.
- Data Format: Data is organized in Brain Imaging Data Structure (BIDS) format and available on the OpenNeuro repository.
Key Insights and Ideas
- Complementarity of EEG and fMRI: The paper strongly advocates for the combination of EEG and fMRI, highlighting how their complementary strengths (temporal vs. spatial resolution) are crucial for robust inner speech decoding. This multimodal approach is presented as a significant step beyond unimodal datasets.
- The Challenge of Inner Speech: Decoding inner speech is presented as a complex but vital challenge for BCI development, particularly for restoring communication for non-verbal individuals.
- Importance of Publicly Available Datasets: The authors emphasize the need for publicly accessible, high-quality datasets to accelerate research in this area, enabling other researchers to build upon their work and develop more effective BCIs.
- Artifact Removal Efficacy: The paper demonstrates the effectiveness of ICA in removing eye artifacts from EEG data, a common challenge in neuroimaging studies. The reduction in correlation between EEG and EOG channels after ICA highlights its utility.
- Neural Correlates of Inner Speech: The fMRI analysis provides insights into the brain regions activated during inner speech, confirming the involvement of language-related areas like Broca's and Wernicke's areas, as well as visual and motor cortices.
Key Results
- Dataset Creation: The primary result is the successful creation and release of the first publicly available bimodal EEG-fMRI dataset for inner speech recognition.
- Artifact Reduction: ICA effectively reduced eye artifacts in the EEG data, as evidenced by visual inspection and a significant decrease in the correlation between EEG and EOG channels.
- fMRI Activation Patterns: Group-level fMRI analysis revealed that inner speech activates language-related regions (Broca's, Wernicke's, supramarginal gyrus, angular gyrus) and visual/motor areas. Social words showed higher activation than number words in specific visual processing areas.
- Individual Subject Consistency: Individual subject fMRI results showed consistent activation patterns, supporting the reliability of the data for subject-specific analyses.
- Low Motion Artifacts: Framewise displacement (FD) analysis indicated low levels of head motion across participants and sessions, suggesting high-quality fMRI data.
Limitations and Future Work
- Small Sample Size: The authors acknowledge that the dataset includes only four participants (after excluding one). They note that while small sample sizes are common in novel research paradigms, larger datasets are needed for broader generalizability.
- Diverse Native Languages: Participants had different native languages, though all were fluent English speakers. The authors cite research suggesting that core language architecture is conserved across languages, but acknowledge this as a factor to consider.
- Non-Simultaneous Acquisition: While chosen to mitigate artifacts and optimize protocols, non-simultaneous acquisition means the EEG and fMRI data are not directly capturing brain activity at the exact same moment.
- Future Work: The paper's aim is to provide a resource for future research. The authors envision this dataset enabling investigations into the advantages of using bimodal versus unimodal data for inner speech recognition and contributing to the development of speech prostheses. Future work could involve developing and testing advanced decoding algorithms using this multimodal dataset.
Summary updated