Single-Stage fMRI-to-3D Reconstruction via Viewpoint-Aware Embedding and Hierarchical Guidance

Xun Zhang, Weihao Xia, Yulong Liu, Bo Yang, Alessandro Bozzon, Pan WangAAAI 2026 · DOI

TL;DR

NeuroSculptor3D is the first single-stage, end-to-end framework capable of reconstructing high-fidelity, textured 3D objects directly from fMRI brain activity. By replacing the common practice of averaging neural features with a "viewpoint-aware" embedding and a hierarchical guidance system, the model achieves state-of-the-art results in both structural accuracy and visual realism.

What Problem Does This Paper Address?

While researchers have made massive strides in decoding 2D images and videos from functional Magnetic Resonance Imaging (fMRI), 3D reconstruction has remained a "final frontier." Previous attempts, such as the MinD-3D model, suffered from three major flaws:

  1. Semantic Dilution: They often averaged brain features across multiple viewpoints, which washed out fine-grained structural details.
  2. Complexity: They relied on cumbersome two-stage training pipelines.
  3. Lack of Detail: The resulting outputs were typically "textureless" grey meshes that lacked the color and surface detail of the original stimuli.

The human brain doesn't just see a flat image; it integrates visual input across multiple perspectives to form an internal 3D representation. This paper aims to emulate that biological process to create more faithful 3D digital twins of what a person is actually seeing.

Methods

The authors introduce NeuroSculptor3D, which consists of three innovative components:

  • Viewpoint-Aware Brain Embedding: Instead of collapsing all neural data into one average feature, the model uses a learnable viewpoint embedding. It concatenates a global brain signal with a specific viewpoint "ID." This allows the model to preserve the spatial variations the brain recording captures as a subject watches an object rotate.
  • Hierarchical Guidance Mechanism: To ensure the reconstructed object makes sense on every level, the model is trained using three parallel "paths":
  • Perceptual Path: Aligns brain features with DINOv2 visual representations to capture fine-grained textures.
  • Semantic Path: Aligns features with CLIP text embeddings to ensure the object belongs to the correct category (e.g., "a chair" vs. "a table").
  • Geometric Path: Maps the signal to a 3D shape autoencoder to enforce structural integrity and volume.
  • TRELLIS Decoder: The framework leverages TRELLIS, a state-of-the-art 3D generative model, as its backbone. This allows the system to take the decoded brain embeddings and synthesize them into a fully textured 3D asset in a single stage.

Datasets

  • fMRI-Shape Dataset: This is the primary benchmark, featuring fMRI recordings from 12 participants viewing 8-second clips of 3D objects (sampled from ShapeNet) rotating 360 degrees.
  • Natural Scenes Dataset (NSD): The researchers used the NSDGENERAL mask from this dataset to identify and isolate the most visually responsive regions of the brain across subjects.

Key Insights and Ideas

  • Viewpoints are the Key: The "Aha!" moment of this paper is that 3D information is encoded in the variation of the brain signal across time/perspective. By explicitly modeling these viewpoints rather than averaging them, the model retains the "3D-ness" inherent in the neural data.
  • Multi-Level Alignment: Reconstructing from fMRI is an "ill-posed" problem (the signal is noisy and indirect). By using hierarchical guidance (semantic, perceptual, and structural), the authors provide enough "priors" to help the model fill in the gaps accurately.
  • Single-Stage Efficiency: Moving to a single-stage process isn't just about speed; it allows the model to learn the relationship between brain activity and 3D geometry more holistically, leading to better alignment than piecemeal two-stage methods.

Key Results

NeuroSculptor3D significantly outperformed previous state-of-the-art models (like MinD-3D and LEA-3D) across all metrics:

  • Semantic Fidelity: In a 10-way classification test (identifying the correct object category), NeuroSculptor3D achieved 63.8% accuracy, compared to MinD-3D’s 48.3%.
  • Structural Accuracy: The model showed a massive improvement in FPD (Frechet Point Cloud Distance), dropping from 8.137 (MinD-3D) to 6.163, indicating a much closer match to the ground-truth geometry.
  • Qualitative Excellence: Visually, the difference is striking. While previous models produced "blobs" that vaguely resembled the target, NeuroSculptor3D produces recognizable, colored, and textured objects (e.g., distinct cars, chairs, and watercraft).

Limitations and Future Work

  • Subject Variability: Like all fMRI-to-visual models, performance drops when testing on a completely new subject (New-Subject New-Category setting) because every brain is wired slightly differently.
  • Dependency on Generative Priors: The quality of the reconstruction is partially capped by the capabilities of the underlying 3D generator (TRELLIS). As 3D generative AI improves, so will this framework.
  • Static Objects: Currently, the model focuses on single, isolated objects. Future work could expand this to complex, multi-object scenes or dynamic environments, moving closer to a true "brain-to-movie" 3D reconstruction.

Summary updated

Abstract

Understanding the neural basis of three-dimensional (3D) perception is a fundamental objective in cognitive neuroscience. Despite advances in decoding 2D visual stimuli from neural data, reconstructing high-fidelity 3D objects with detailed texture and geometry remains largely unexplored. In this work, we introduce NeuroSculptor3D, the first single-stage, end-to-end framework for reconstructing textured 3D shapes directly from brain activity. NeuroSculptor3D integrates a viewpoint-aware brain embedding module that captures fine-grained spatial variations across visual perspectives, and a hierarchical guidance mechanism that aligns brain-derived features with perceptual, semantic, and structural priors. Together, these components facilitate the generation of consistent multi-view embeddings, which are then decoded via TRELLIS to produce high-quality textured 3D reconstructions. Experiments on the fMRI-Shape dataset demonstrate that NeuroSculptor3D outperforms existing baselines across multiple settings, achieving significant improvements in both structural accuracy and semantic consistency. Code will be released to facilitate further research.
BibTeX
@inproceedings{zhang2026single,
  author = {Xun Zhang and Weihao Xia and Yulong Liu and Bo Yang and Alessandro Bozzon and Pan Wang},
  title = {{Single-Stage fMRI-to-3D Reconstruction via Viewpoint-Aware Embedding and Hierarchical Guidance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i21.38864},
}
Figure