A synchronized multimodal neuroimaging dataset for studying brain language processing

Shaonan Wang, Xiaohan Zhang, Jiajun Zhang, Chengqing ZongScientific 2022 · DOI

Abstract

We present a synchronized multimodal neuroimaging dataset for studying brain language processing (SMN4Lang) that contains functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data on the same 12 healthy volunteers while the volunteers listened to 6 hours of naturalistic stories, as well as high-resolution structural (T1, T2), diffusion MRI and resting-state fMRI data for each participant. We also provide rich linguistic annotations for the stimuli, including word frequencies, syntactic tree structures, time-aligned characters and words, and various types of word and character embeddings. Quality assessment indicators verify that this is a high-quality neuroimaging dataset. Such synchronized data is separately collected by the same group of participants first listening to story materials in fMRI and then in MEG which are well suited to studying the dynamic processing of language comprehension, such as the time and location of different linguistic features encoded in the brain. In addition, this dataset, comprising a large vocabulary from stories with various topics, can serve as a brain benchmark to evaluate and improve computational language models.

Introduction

Understanding the neural mechanisms of human language requires capturing brain activity with both high spatial and temporal resolution. Historically, researchers have faced a "modality gap": Functional Magnetic Resonance Imaging (fMRI) provides excellent spatial localization but lacks the temporal speed to track speech processing, while Magnetoencephalography (MEG) offers millisecond-level temporal resolution but limited spatial precision. Furthermore, most existing datasets collect these modalities from different participant groups or use artificial "toy" stimuli rather than naturalistic speech.

The SMN4Lang dataset addresses these limitations by providing synchronized fMRI and MEG data from the same 12 individuals listening to 6 hours of naturalistic stories. This "same brain" approach allows for multimodal fusion, enabling researchers to map the "where" and "when" of language comprehension within a unified framework.

Experimental Design and Methods

The study involved 12 healthy, right-handed Mandarin speakers. The experimental protocol was divided into two main phases:

  1. Sequential Multimodal Imaging: Participants listened to 60 naturalistic stories (4–7 minutes each) twice—once in an fMRI scanner and once in an MEG array. To minimize memory effects, sessions were separated by at least one month.
  2. Naturalistic Stimuli: The stories covered diverse topics, including education and culture, providing a rich linguistic context far beyond isolated sentences.
  3. Task Validation: To ensure sustained attention, participants completed four-choice comprehension quizzes after each story.
Figure 1 & 2: (Top) Schematic overview of the study procedure including fMRI/MEG tasks and structural MRI. (Bottom) Characteristics of the 60 story stimuli, showing the relationship between duration and the number of sentences/words.

The SMN4Lang Dataset

Hosted on OpenNeuro (ds004078), the dataset comprises several key components:

  • Neuroimaging Data: Includes raw and preprocessed BOLD fMRI, MEG, structural MRI (T1, T2), diffusion MRI for white matter tractography, and resting-state fMRI.
  • Linguistic Annotations: The stimuli are densely annotated with manual transcriptions, time-aligned word/character boundaries, part-of-speech (POS) tags, and manually verified syntactic dependency trees.
  • Computational Embeddings: To facilitate AI research, the dataset includes representations from Word2Vec, BERT (12 layers), and GPT-2 (25 layers), allowing for direct comparison between AI models and human brain activity.

Quality Assessment and Validation

The dataset underwent rigorous quality checks. Participants demonstrated high engagement, with mean quiz accuracies exceeding 91% across both fMRI and MEG sessions. Technical metrics also confirmed high data fidelity:

  • Motion Control: The average Framewise Displacement in fMRI was only 3.11%, indicating exceptionally clean data for long-duration scans.
  • Neural Tracking: Encoding models using BERT embeddings successfully predicted fMRI signals in the core language network, including the temporal, parietal, and frontal lobes.
  • MEG Consistency: Inter-subject correlation (ISC) analysis showed robust neural tracking of speech rhythms, particularly in the delta (1–4Hz) and theta (4–8Hz) bands, which correspond to phrase and syllable structures.

Discussion and Applications

The SMN4Lang dataset serves as a powerful tool for both neuroscience and artificial intelligence. By using fMRI to constrain MEG source estimates, researchers can create high-resolution 4D models of language processing. Additionally, the dataset acts as a "brain benchmark" for Natural Language Processing (NLP). For instance, researchers found that middle layers of BERT (specifically layer 7) correlate most strongly with brain activity, suggesting these layers capture critical semantic and syntactic integrations.

Limitations and Future Work

While comprehensive, the dataset is limited by its sample size (12 participants) and its focus on Mandarin Chinese. Future work should aim to expand this multimodal framework to larger cohorts and diverse language families to uncover universal neural mechanisms of language. Additionally, while the month-long gap between sessions mitigates repetition effects, the non-simultaneous collection means the participant's internal state may vary between modalities.

Conclusion

SMN4Lang provides a rare, high-fidelity look at the human brain in action. By bridging the gap between spatial and temporal imaging within the same individuals, it offers a gold standard for studying the dynamic complexities of natural language comprehension.

Summary updated

Abstract

We present a synchronized multimodal neuroimaging dataset for studying brain language processing (SMN4Lang) that contains functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data on the same 12 healthy volunteers while the volunteers listened to 6 hours of naturalistic stories, as well as high-resolution structural (T1, T2), diffusion MRI and resting-state fMRI data for each participant. We also provide rich linguistic annotations for the stimuli, including word frequencies, syntactic tree structures, time-aligned characters and words, and various types of word and character embeddings. Quality assessment indicators verify that this is a high-quality neuroimaging dataset. Such synchronized data is separately collected by the same group of participants first listening to story materials in fMRI and then in MEG which are well suited to studying the dynamic processing of language comprehension, such as the time and location of different linguistic features encoded in the brain. In addition, this dataset, comprising a large vocabulary from stories with various topics, can serve as a brain benchmark to evaluate and improve computational language models.
BibTeX
@inproceedings{wang2022synchronized,
  author = {Shaonan Wang and Xiaohan Zhang and Jiajun Zhang and Chengqing Zong},
  title = {{A synchronized multimodal neuroimaging dataset for studying brain language processing}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41597-022-01708-5},
}
Figure