Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding

Mohammad Moradi, Morteza Moradi, Marco Grassia, Giuseppe MangioniarXiv 2026 · arXiv

TL;DR

Brain-GraSP is a novel, lightweight framework that improves fMRI-to-image brain decoding by extracting spatial saliency maps and textual cues directly from brain signals using Graph Neural Networks. By conditioning a single, frozen Stable Diffusion model with these structural and semantic priors, it reconstructs images from brain activity with unprecedented object conformity and spatial accuracy.

What Problem Does This Paper Address?

While recent advances in deep learning have made it possible to reconstruct images that a person is viewing based purely on their fMRI brain scans, these systems still face a major hurdle: they struggle with structure. Existing visual brain decoding (VBD) models often misplace objects, get the count wrong, or fail to preserve the basic spatial layout of the original scene.

Furthermore, many state-of-the-art methods rely on computationally heavy, multi-stage diffusion processes that are expensive to train and run. The authors of Brain-GraSP set out to fix both issues by introducing a "green AI" approach that focuses on saliency—identifying exactly where and what the most important objects are in a scene before generating the image.

Methods

Instead of training a massive end-to-end model from scratch, Brain-GraSP cleverly builds on top of precomputed fMRI-CLIP embeddings from a previous landmark model called MindEye.

The architecture consists of three main pillars:

  1. GNN-based Saliency Modeling: The framework treats fMRI-CLIP embedding tokens as nodes in a graph. A Graph Neural Network (specifically GraphSAGE) analyzes the spatial and semantic proximity of these tokens to predict node-level importance weights, which are then mapped into a 2D spatial saliency mask.
  2. Semantic Cue Extraction: Simultaneously, the embeddings are projected into CLIP’s text-aligned space to extract language-grounded textual descriptors (e.g., names of objects in the scene).
  3. Saliency-Guided Image Generation: These spatial masks and textual cues condition a completely frozen Stable Diffusion model via an IP-Adapter. Saliency is applied during generation using three distinct modes: image inpainting, foreground/background mask-blending, and candidate ranking based on structural and semantic alignment scores.
The architecture of Brain-GraSP for fMRI-based visual reconstruction with saliency priors and textual cues.

Datasets

The model was trained and evaluated using the benchmark Natural Scenes Dataset (NSD), focusing on fMRI recordings from subjects 1, 2, 5, and 7. Because the NSD dataset lacks human-annotated saliency masks, the authors used an external model (EDN) to generate pseudo-ground truth saliency maps for the last 301 images of the test set to train their GNN. Evaluation was strictly performed on the remaining 681 images.

Key Insights and Ideas

  • Targeting Saliency Early: Most prior methods analyze saliency post-hoc on already generated images. Brain-GraSP extracts saliency directly from the initial brain embeddings. Because these embeddings precede the stochastic, artifact-prone diffusion process, they provide a cleaner, more faithful representation of the original brain activity.
  • Graphs for Brain Signals: Modeling fMRI-CLIP tokens as a graph naturally captures the relational and structural dependencies inherent in visual perception. GraphSAGE proved particularly effective here due to its inductive neighborhood aggregation, which excels at modeling distributed patterns.
  • Modular & Sustainable Design: By utilizing a frozen generative model and precomputed embeddings, Brain-GraSP demonstrates that researchers don't need to reinvent the wheel (or burn massive compute) to achieve state-of-the-art results.

Key Results

Quantitatively, Brain-GraSP outperformed top-tier baselines like MindEye, MindBridge, and BOI across several critical metrics assessing structural fidelity and semantic alignment:

  • Pixel Correlation (PixCorr): Reached 0.326, outperforming MindEye's 0.305.
  • Structural Similarity (SSIM): Achieved 0.299, a notable step up from baselines.
  • CLIP Semantic Alignment: Scored 98.80%, showing exceptional conceptual accuracy.
Table 1. Quantitative evaluation of Brain-GraSP against three state-of-the-art models (top), along with a subject-wise perfor- mance breakdown. Highlighted scores indicate better performance.

Qualitatively, the results speak for themselves. As shown below, Brain-GraSP preserves the distinct layout and object boundaries of the original stimuli far more reliably than competing models.

Fig. 2. Qualitative examples of reconstructions by our model, MindEye, MindBridge and BOI.

Ablation studies confirmed that removing either the saliency prior or the text cues caused a sharp drop in performance, proving that both spatial and semantic constraints are vital for accurate decoding.

Limitations and Future Work

The primary limitation acknowledged by the authors is their reliance on an AI model (EDN) to generate pseudo-ground truth saliency maps for training. This introduces the AI model's own biases and doesn't perfectly mirror true human visual attention.

Additionally, while Brain-GraSP dominates higher-level perceptual and semantic metrics, it slightly underperforms MindEye on low-to-mid-level features (like AlexNet-2/5 scores). Future work will likely focus on refining the GNN architectures and incorporating actual human eye-tracking data to better align the saliency priors with real human vision.

Summary updated

Abstract

Recent progress in brain-guided image generation has improved the quality of fMRI-based reconstructions; however, fundamental challenges remain in preserving object-level structure and semantic fidelity. Many existing approaches overlook the spatial arrangement of salient objects, leading to conceptually inconsistent outputs. We propose a saliency-driven decoding framework that employs graph-informed saliency priors to translate structural cues from brain signals into spatial masks. These masks, together with semantic information extracted from embeddings, condition a diffusion model to guide image regeneration, helping preserve object conformity while maintaining natural scene composition. In contrast to pipelines that invoke multiple diffusion stages, our approach relies on a single frozen model, offering a more lightweight yet effective design. Experiments show that this strategy improves both conceptual alignment and structural similarity to the original stimuli, while also introducing a new direction for efficient, interpretable, and structurally grounded brain decoding.
BibTeX
@misc{moradi2026brain,
  author = {Mohammad Moradi and Morteza Moradi and Marco Grassia and Giuseppe Mangioni},
  title = {{Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding}},
  year = {2026},
  eprint = {2604.10617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.10617},
}

Loading related content...

Figure