BOLD5000: A public fMRI dataset of 5000 images

Nadine Chang, John A. Pyles, Abhinav Gupta, Michael J. Tarr, Elissa M. AminoffarXiv 2018 · DOI · OpenReview

Abstract

Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches. Yet, human neuroimaging studies of visual perception still rely on small numbers of images (around 100) due to time-constrained experimental procedures. To apply statistical learning approaches that integrate neuroscience, the number of images used in neuroimaging must be significantly increased. We present BOLD5000, a human functional MRI (fMRI) study that includes almost 5,000 distinct images depicting real-world scenes. Beyond dramatically increasing image dataset size relative to prior fMRI studies, BOLD5000 also accounts for image diversity, overlapping with standard computer vision datasets by incorporating images from the Scene UNderstanding (SUN), Common Objects in Context (COCO), and ImageNet datasets. The scale and diversity of these image datasets, combined with a slow event-related fMRI design, enable fine-grained exploration into the neural representation of a wide range of visual features, categories, and semantics. Concurrently, BOLD5000 brings us closer to realizing Marr's dream of a singular vision science - the intertwined study of biological and computer vision.

Introduction: Breaking the Small Data Bottleneck

For decades, vision science has been divided between biological vision (neuroscience) and computer vision (AI). While computer vision has been revolutionized by "big data" benchmarks like ImageNet, human neuroimaging has remained constrained by a "small data" bottleneck. Most fMRI studies utilize approximately 100 images because scanning time is expensive and participant endurance is limited. This small sample size hinders the application of modern statistical learning techniques and prevents a robust comparison between human brain activity and the complex, multi-layered processing of Deep Convolutional Neural Networks (CNNs). BOLD5000 was designed to overcome this limitation, aiming to unify biological and artificial vision research.

The researchers employed a slow event-related fMRI design. Unlike "fast" designs where stimuli overlap in time, the slow design presents an image for 1 second followed by a 9-second fixation period. This allows for the isolation of the specific blood-oxygen-level-dependent (BOLD) signal for every individual image trial.

Key technical aspects include:

  • Deep Imaging: Rather than scanning a large cohort briefly, the study focused on four participants scanned extensively—roughly 20 hours of MRI time each across 15-16 sessions.
  • Gray World Normalization: To ensure brain responses were driven by semantic content rather than low-level luminance, all images were adjusted to have uniform brightness.
  • Validation Techniques: Representational Similarity Analysis (RSA) was used to compare brain activity patterns to the internal layers of AlexNet, and t-SNE was used to visualize how the brain clusters different image types.

Datasets: A Bridge to Computer Vision Benchmarks

The BOLD5000 dataset consists of 4,916 unique images selected from three "gold standard" computer vision benchmarks to ensure maximum overlap with AI research:

  1. Scene UNderstanding (SUN): 1,000 images of indoor and outdoor environments.
  2. Common Objects in Context (COCO): 2,000 images of objects in complex, real-world settings.
  3. ImageNet: 1,916 images of centered, singular objects.
Sample images from the SUN, COCO, and ImageNet datasets used in BOLD5000.

The final dataset includes 5,254 total trials per participant, including repeated images to assess reliability. Analysis of the COCO subset shows that it maintains a representative distribution of object instances compared to the full COCO 2014 training set, ensuring the complexity of real-world scenes is preserved.

Distribution of object instances in the COCO subset compared to the full COCO 2014 training set.

Key Insights: Hierarchical Alignment and Latent Spaces

  • Hierarchical Alignment: The study confirmed a shared hierarchical structure between the human visual system and CNNs. "Lower" brain areas (e.g., early visual cortex) correlate with early CNN layers, while "higher" areas (e.g., Parahippocampal Place Area) correlate with deeper, semantic layers.
  • The Data Challenge: The authors argue that the primary hurdle to integrating AI and neuroscience is the lack of shared data, not a lack of theory. BOLD5000 serves as a "Rosetta Stone" for these fields.
  • Visualizing Latent Space: t-SNE visualization of fMRI data revealed that high-level visual areas naturally cluster images by category (e.g., "Food" vs. "Living Animate") without explicit supervision, mirroring AI organization.

Key Results: Scale and Reliability

  • Scale: BOLD5000 is an order of magnitude larger than any previous slow event-related fMRI dataset.
  • Consistency: RSA heatmaps demonstrated remarkably consistent results across all participants and both brain hemispheres.
  • Reliability: The correlation of brain activity for repeated images was significantly higher than for different images, proving the signal's robustness despite the massive stimulus set.
  • Open Access: The complete dataset, including raw DICOM files and stimulus images, is publicly available.

Limitations and Future Outlook

The primary limitation is the small participant pool (N=4). While the data for these individuals is exceptionally deep, it may not capture the full range of human neurological diversity. Additionally, while 5,000 images is a major advancement for neuroscience, it remains small compared to the millions of images used in AI training. The authors suggest that the future lies in "stitching" together deep-imaging datasets from multiple laboratories to eventually reach the scale of ImageNet for the human brain.

Summary updated

Abstract

Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches. Yet, human neuroimaging studies of visual perception still rely on small numbers of images (around 100) due to time-constrained experimental procedures. To apply statistical learning approaches that integrate neuroscience, the number of images used in neuroimaging must be significantly increased. We present BOLD5000, a human functional MRI (fMRI) study that includes almost 5,000 distinct images depicting real-world scenes. Beyond dramatically increasing image dataset size relative to prior fMRI studies, BOLD5000 also accounts for image diversity, overlapping with standard computer vision datasets by incorporating images from the Scene UNderstanding (SUN), Common Objects in Context (COCO), and ImageNet datasets. The scale and diversity of these image datasets, combined with a slow event-related fMRI design, enable fine-grained exploration into the neural representation of a wide range of visual features, categories, and semantics. Concurrently, BOLD5000 brings us closer to realizing Marr's dream of a singular vision science - the intertwined study of biological and computer vision.
BibTeX
@misc{chang2018bold,
  author = {Nadine Chang and John A. Pyles and Abhinav Gupta and Michael J. Tarr and Elissa M. Aminoff},
  title = {{BOLD5000: A public fMRI dataset of 5000 images}},
  year = {2018},
  eprint = {1809.01281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.01281},
}
Figure