TL;DR
This paper introduces CNeuroMod-THINGS, a novel, densely-sampled fMRI dataset designed to advance neuro-AI modeling of human visual perception. By combining the rich image stimuli of the THINGS initiative with the extensive fMRI data from the CNeuroMod project, this dataset offers unprecedented opportunities to study neural representations of semantic concepts in both controlled and naturalistic settings.
What Problem Does This Paper Address?
The field of neuro-AI is increasingly reliant on large-scale neuroimaging datasets to train and validate models that can predict brain activity. However, existing datasets often lack sufficient sampling density (i.e., the number of scans per participant) or comprehensive semantic coverage of visual stimuli. This limits the ability to capture fine-grained neural representations and build robust, individual-specific models of human vision. CNeuroMod-THINGS aims to bridge this gap by providing a dataset with a high number of fMRI sessions per participant, coupled with a well-annotated and semantically diverse set of visual stimuli.
Methods
The CNeuroMod-THINGS dataset was created by having four participants from the CNeuroMod project undergo 33-36 fMRI sessions each. During these sessions, participants performed a continuous image recognition task, viewing 4320 images from the THINGS stimulus set, which spans 720 object categories. This dense sampling allows for data-driven analyses at the single-image level. Standard fMRI preprocessing pipelines (fMRIPrep) were used, and single-trial responses were estimated using the GLMsingle toolbox. Additional methods included eye-tracking for fixation compliance, population receptive field (pRF) mapping, and functional localizer (fLoc) tasks to define category-selective regions of interest (ROIs).
Datasets
- CNeuroMod-THINGS: The primary dataset introduced in this paper. It comprises fMRI data, behavioral responses, and eye-tracking data from four participants who completed 33-36 sessions each.
- THINGS (The Handcrafted Image Stimuli): Provided the set of 4320 naturalistic images spanning 720 object categories, along with extensive annotations and behavioral data.
- CNeuroMod: Provided the participants and the infrastructure for acquiring hundreds of hours of fMRI data across various tasks.
- Natural Scenes Dataset (NSD): Mentioned as a comparable large fMRI dataset for naturalistic images.
- BOLD5000: Another large fMRI dataset for visual stimuli.
Key Insights and Ideas
- Dense Sampling for Single-Image Analysis: The high number of sessions per participant is crucial for enabling robust, data-driven analyses at the level of individual images, which is a significant advancement over datasets with fewer sessions.
- Bridging Controlled and Naturalistic Vision: By using well-characterized images in a controlled recognition task, the dataset bridges the gap between studies of vision in naturalistic settings (like movie watching) and more traditional, controlled visual paradigms.
- Individualized Brain Models: The dense sampling and high-quality data are expected to facilitate the development of more accurate and personalized computational models of human visual processing.
- Synergy of Existing Resources: The project demonstrates the power of combining existing, large-scale datasets and projects (THINGS and CNeuroMod) to create novel, high-value resources.
Key Results
- High Response Rates: Participants exhibited high response rates (91.13% to 99.84%) during the recognition task, indicating good engagement.
- Low Motion: Framewise displacement (FD) values were generally low across participants, with sub-03 showing exceptionally low motion, contributing to high noise ceilings.
- Consistent Fixation: Eye-tracking data revealed high levels of fixation compliance, with participants generally staying within a small visual angle from the fixation marker.
- Robust Memory Effects: Behavioral data showed clear evidence of memory effects, with better recognition for images repeated within a session compared to between sessions.
- Stimulus-Specific Signals: Noise ceiling analyses indicated consistent stimulus-specific signals in visual cortical areas, including those with known categorical preferences (FFA, EBA, PPA).
- t-SNE Reveals Semantic Structure: Dimensionality reduction analyses (t-SNE) on fMRI data successfully visualized the semantic organization of visual representations within category-selective ROIs.
Limitations and Future Work
- Limited Number of Participants: The dataset currently includes data from only four participants. While densely sampled, this limits generalizability to the broader population. Future work could involve expanding the participant pool.
- Variability in Memory Effects: The paper notes high variability in memory effects across runs and sessions, suggesting that further modeling is needed to account for these sources of variability.
- Future Modeling: The authors envision the dataset being used to train and test a wide range of neuro-AI models, from simple linear models to complex deep neural networks, to better understand how the brain represents visual information. The dataset's structure is designed to support such diverse modeling efforts.
Summary updated