fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding

Xuelin Qian, Yun Wang, Jingyang Huo, Xinwei Sun, Yanwei Fu, Jianfeng FengSubmission to ICLR 2024 · Rejected

TL;DR

Current fMRI visual decoding models struggle to generalize across people due to structural anatomical differences and small paired datasets. fMRI-PTE solves this by projecting native fMRI signals into unified 2D cortical surface flatmaps and pre-training a two-stage autoencoder on over 40,000 subjects from the UK Biobank. When paired with generative models like Stable Diffusion or MaskGIT, fMRI-PTE sets a strong new standard for zero-shot and few-shot cross-subject brain decoding.

What Problem Does This Paper Address?

Decoding visual stimuli from human brain activity via functional Magnetic Resonance Imaging (fMRI) has long been constrained to single-subject models. Because human brains exhibit substantial anatomical and functional variability, voxel coordinates and feature dimensions do not directly map from one person to another. A model trained to reconstruct images from Subject A’s brain scans typically fails when tested on Subject B.

Moreover, fMRI-image paired datasets are notoriously small (usually fewer than ten subjects), leading to overfitting and blurry reconstructions. While foundation models (like BERT or MAE) have revolutionized vision and NLP, applying them to multi-subject neuroimaging has been hindered by inconsistent data formats, loss of spatial structure when flattening voxels into 1D vectors, and the loss of high-frequency cortical signals during compression.

Methods

fMRI-PTE addresses these challenges through a two-part framework:

  1. Unified 2D Cortical Surface Representation: Instead of flattening fMRI voxels into 1D vectors, the authors map native 3D fMRI time series onto a standardized group surface space (32k_fs_LR). Using pycortex, cortical activation flatmaps are rendered as standardized $1023 \times 2514$ 2D images. This preserves spatial topology and local neighborhood structures across subjects.
  2. Two-Stage Autoencoding:
  3. Stage 1 (Quantized Reconstruction): A convolutional VQGAN-style encoder–decoder ($E_1, D_1$) learns to discretize the continuous 2D surface maps into discrete codebook indices. Convolutions preserve local high- and low-frequency signal textures that standard vision transformers often blur.
  4. Stage 2 (Transformer Compression): Operating directly on the discrete indices, a Vision Transformer encoder ($E_2$) models global, long-range dependencies across distant cortical regions. A mapping bottleneck compresses the representations (achieving a ~78% compression ratio, down to $77 \times 768$), while a masked transformer decoder ($D_2$) reconstructs the original indices conditioned on the compressed latent vector.
  5. Downstream Decoding: The compressed latent representation is mapped via linear ridge regression to conditional generative decoders—specifically Stable Diffusion (SD) or MaskGIT (MG)—to synthesize the visual stimuli seen by the subject.
fMRI Volumes ➔ 32k_fs_LR Surface ➔ 2D Flatmap ➔ [Stage 1: VQ-Autoencoder] ➔ [Stage 2: ViT Compressor] ➔ Diffusion/MaskGIT ➔ Decoded Image

Datasets

  • UK Biobank (UKB): Pre-training dataset comprising resting-state fMRI from ~39,630 training subjects and 1,000 held-out validation subjects (each with 490 volumes per session).
  • Natural Scenes Dataset (NSD): Benchmark for downstream evaluation consisting of 8 subjects viewing natural images from MS-COCO. The authors evaluated models using the 982 shared test images across Subjects 1, 2, 5, and 7.

Key Insights and Ideas

  • fMRI surface flatmaps act like natural textures, not photos: Unlike natural images, brain surface activations exhibit lower spatial redundancy and carry critical information in high-frequency local textures. Standard masked autoencoding (MAE) applied directly to fMRI often causes regional blurring; decoupling local texture quantization (via CNNs) from global cross-region modeling (via Transformers) prevents this signal degradation.
  • True cross-subject transfer is achievable without subject-specific retraining: By mapping to a common 2D surface space before pre-training on 40,000 individuals, the encoder learns a shared functional latent space. A model trained on three subjects can successfully decode fMRI scans from an unseen fourth subject.

Key Results

  • fMRI Signal Reconstruction: On NSD cross-subject evaluation, fMRI-PTE achieved a Pearson correlation of 0.8546 (matching the 0.8547 upper bound of the Stage 1 autoencoder), an SSIM of 0.7937, and an MSE of 0.0263, outperforming MAE (Pearson: 0.8439) and MBM (Pearson: 0.7372).
  • One-to-One Cross-Subject Decoding: When trained on one subject and tested on another (e.g., $7 \rightarrow 1$), fMRI-PTE paired with MaskGIT achieved 72.82% AlexNet and 63.21% CLIP accuracy, comfortably outperforming MinD-Vis (59.67% AlexNet / 56.71% CLIP) and LEA (70.39% AlexNet / 60.89% CLIP).
  • Many-to-One Decoding: In leave-one-subject-out transfer settings (e.g., aligning on subjects 2, 5, 7 and evaluating zero-shot on subject 1), fMRI-PTE (MG) attained 76.66% AlexNet and 66.83% CLIP scores.
  • Within-Subject Decoding: fMRI-PTE achieved competitive or state-of-the-art results on NSD (e.g., 88.59% AlexNet top-5, 84.09% Inception), surpassing several multimodal baselines that utilize both text and vision conditioning.

Limitations and Future Work

  • Computational Demands: Pre-training on 40,000 subjects requires substantial compute (reported at ~6 days across 8 NVIDIA A100 GPUs).
  • Simple Downstream Alignment: Downstream fMRI-to-image feature alignment currently relies primarily on linear ridge regression. End-to-end multi-modal fine-tuning could yield richer cross-modal representations.
  • ROI Selection: Downstream visual decoding experiments were restricted to 8,405 visual cortical vertices (HCP-MMP atlas), leaving non-visual cognitive decoding tasks for future exploration.

Summary updated

Abstract

The exploration of brain activity and its decoding from fMRI data has been a longstanding pursuit, driven by its potential applications in brain-computer interfaces, medical diagnostics, and virtual reality. Previous approaches have primarily focused on individual subject analysis, highlighting the need for a more universal and adaptable framework, which is the core motivation behind our work. In this work, we propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training, with a focus on addressing the challenges of varying fMRI data dimensions due to individual brain differences. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving distinct brain activity patterns. We introduce a novel learning strategy tailored for pre-training 2D fMRI images, enhancing the quality of reconstruction. fMRI-PTE's adaptability with image generators enables the generation of well-represented fMRI features, facilitating various downstream tasks, including within-subject and cross-subject brain activity decoding. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach. Extensive experiments validate and support our claims, offering a promising foundation for further research in this domain.
BibTeX
@misc{qian2023fmri,
  author = {Xuelin Qian and Yun Wang and Jingyang Huo and Xinwei Sun and Yanwei Fu and Jianfeng Feng},
  title = {{fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding}},
  year = {2023},
  eprint = {2311.00342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.00342},
}
Figure