Alljoined - A dataset for EEG-to-Image decoding

Jonathan Xu, Bruno Aristimunha, Max Emanuel Feucht, Emma Qian, Charles Liu, Tazik Shahjahan, Martyna Spyra, Steven Zifan Zhang, Nicholas Short, Jioh Kim, Paula Perdomo, Ricky Renfeng Mao, Yashvir Sabharwal, Michael Ahedor Moaz Shoura, Adrian NestorarXiv 2024 · DOI · arXiv

Introduction

The Alljoined1 dataset represents a significant advancement in the field of neural decoding, specifically targeting the reconstruction of visual stimuli from electroencephalography (EEG) signals. While functional Magnetic Resonance Imaging (fMRI) has historically dominated this domain due to its superior spatial resolution, its practical limitations—high cost, lack of portability, and low temporal resolution—necessitate the exploration of EEG-based alternatives. EEG offers millisecond-level temporal resolution, providing a more granular view of the brain's dynamic response to visual information.

A primary challenge in EEG-to-image decoding has been the scarcity of large-scale, high-quality datasets. Many existing benchmarks suffer from limited class diversity or experimental designs that allow for information leakage. Alljoined1 addresses these gaps by providing a massive, open-source repository of brain responses to 10,000 natural images per participant, matching the scale of modern fMRI benchmarks.

Experimental Methodology

To ensure a high signal-to-noise ratio (SNR), the researchers implemented a rigorous experimental pipeline. A critical departure from previous studies is the use of a 300ms stimulus presentation window, rather than the typical 50–100ms. This extended duration allows for both initial perceptual encoding and subsequent higher-level semantic processing. To minimize predictive noise, stimuli were presented in a randomized order with a variable inter-trial jitter of 0–50ms.

Figure 1: The experimental design features a hierarchical structure of sessions and blocks, utilizing a 300ms presentation window and randomized trial ordering to maximize data quality.

Data acquisition utilized a 64-electrode BioSemi ActiveTwo system. The preprocessing workflow included Independent Component Analysis (ICA) for ocular artifact removal and the AUTOREJECT algorithm for automated handling of noisy segments.

Dataset Characteristics

The Alljoined1 dataset comprises data from eight healthy participants. Each participant viewed 10,000 natural images sourced from the Natural Scenes Dataset (NSD) and MS-COCO. The image set includes 9,000 unique images per subject and 1,000 shared images to facilitate cross-subject analysis. The current release includes 46,080 epochs of brain responses, covering a diverse range of supercategories including people, animals, and household objects.

Analysis and Key Results

Analysis of the dataset confirms the capture of high-fidelity visual signals. Event-Related Potentials (ERPs) demonstrate consistent neural activity beginning at 150ms and peaking between 250–300ms. This timing aligns with the expected activation of the visual cortex. Topographic heatmaps further validate the data, showing localized activation in the occipital and parietal regions.

Figure 2: Topographic maps and ERP waveforms illustrate the temporal and spatial progression of neural activity, with significant peaks occurring between 250ms and 300ms post-stimulus.

The authors highlight the "300ms sweet spot" as essential for capturing rich neural dynamics. Furthermore, by calculating the Standardized Measurement Error (SME), they demonstrate that multi-session data collection, while increasing variance, results in a more robust and authentic representation of neural signals compared to single-session snapshots.

Discussion and Future Directions

Despite its scale, the dataset has limitations, including a demographic skew (6 male, 2 female; average age 22). Future iterations could benefit from higher-density electrode arrays focused on the visual cortex. Additionally, extending these decoding techniques from perceived images to mental imagery remains a key objective for advancing brain-computer interface (BCI) technologies.

Summary updated

BibTeX
@misc{xu2024alljoined,
  author = {Jonathan Xu and Bruno Aristimunha and Max Emanuel Feucht and Emma Qian and Charles Liu and Tazik Shahjahan and Martyna Spyra and Steven Zifan Zhang and Nicholas Short and Jioh Kim and Paula Perdomo and Ricky Renfeng Mao and Yashvir Sabharwal and Michael Ahedor Moaz Shoura and Adrian Nestor},
  title = {{Alljoined - A dataset for EEG-to-Image decoding}},
  year = {2024},
  eprint = {2404.05553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05553},
}
Figure