SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance

Minghan Yang, Lan Yang, Ke Li, Honggang Zhang, Kaiyue Pang, Yizhe Song

TL;DR

SemVideo introduces a novel hierarchical semantic guidance framework to reconstruct high-quality, temporally coherent videos directly from fMRI brain activity. By decoding brain signals into three distinct levels of textual descriptions—static anchors, motion narratives, and holistic summaries—it bridges the gap between slow blood-flow brain responses and rapid visual transitions, setting a new state-of-the-art in neural decoding.

What Problem Does This Paper Address?

Reading minds to reconstruct what someone is watching is one of the most exciting frontiers in cognitive neuroscience. While deep learning models (especially diffusion models) have gotten remarkably good at reconstructing static images from functional Magnetic Resonance Imaging (fMRI) data, video reconstruction remains a massive hurdle.

The core challenge lies in a fundamental mismatch of timescales. fMRI relies on the blood-oxygen-level-dependent (BOLD) signal, which tracks slow hemodynamic changes in the brain over several seconds. Videos, on the other hand, change at dozens of frames per second. Because of this, current fMRI-to-video models suffer from two major flaws:

  1. Appearance Mismatch: Salient objects morph inconsistently from frame to frame (e.g., a cat suddenly changing color or turning into a dog).
  2. Motion Misalignment: The reconstructed video lacks temporal coherence, resulting in jittery, unrealistic, or abrupt motion transitions.

Inspired by neuroscience research showing that the human brain perceives video discretely via keyframes rather than processing every single pixel continuously, the authors set out to guide video generation using hierarchical text descriptions decoded straight from the brain.

Methods

The SemVideo framework consists of two main phases: semantic mining during training, and a multi-stage decoding and rendering pipeline during inference.

1. SemMiner (Semantic Mining)

Instead of relying on basic image-captioning models that describe frames in isolation, the authors designed SemMiner. Built on top of a Multimodal Large Language Model (MLLM), it decomposes video stimuli into three tiers of text:

  • Anchor Descriptions ($C_{anchor}$): Captures the static visual details of the very first frame to anchor the scene's appearance.
  • Motion-oriented Narratives ($C_{motion}$): Focuses purely on actions, speeds, and transitions.
  • Holistic Summaries ($C_{holi}$): Integrates both static and dynamic elements into a global overview.

To prevent the MLLM from hallucinating, SemMiner uses a two-stage "prompt chain" where it first generates a strict, 20-word basic summary to act as a constraint (or "rein") before diving into the detailed descriptions.

2. The Decoding Pipeline

During reconstruction, SemVideo uses two specialized decoders to process the fMRI data:

  • Semantic Alignment Decoder (SAD): Maps fMRI signals into the CLIP text embedding space for the three semantic targets ($C_{anchor}$, $C_{motion}$, $C_{holi}$). It utilizes a causal transformer called a Refineformer to maximize the extraction of meaningful neural activity while filtering out fMRI noise.
  • Motion Adaptation Decoder (MAD): Reconstructs the dynamic motion patterns. It features a novel tripartite attention fusion architecture that blends spatial self-attention, temporal self-attention, and semantic-guided cross-attention (conditioned on the decoded motion narrative).
Figure 2. Overview of the SemVideo training pipeline. In the first stage, the Semantic Alignment Decoder is trained to map fMRI signals to three levels of semantic targets, denoted as Z(CL). In the second stage, the Motion Adaptation Decoder is trained to utilize the predicted motion semantics ˆZ(Cmotion) to refine the latent embedding of each reconstructed frame.

3. Conditional Video Render (CVR)

Finally, the decoded cues are progressively combined. The motion latents generate a baseline blurry video. The anchor feature refines the first frame using a Text-to-Image model, and a final Text-to-Video model synthesizes the final coherent video using the anchor frame, the blurry motion sequence, and the holistic summary.

Datasets

The model was trained and evaluated on two prominent fMRI-video datasets:

  • CC2017: Contains fMRI recordings from three subjects watching 23 high-definition natural movie clips.
  • CC2017-SE (Semantic Extension): A new benchmark introduced by the authors. They applied SemMiner to enrich the original CC2017 dataset with multi-level textual descriptions, making it openly available for future research.
  • HCP 7T: A high-field (7 Tesla) fMRI subset from the Human Connectome Project, featuring three subjects watching movie clips.

Key Insights and Ideas

  • Text as a Regularizer: Mapping fMRI directly to raw video pixels or latents is incredibly noisy. Using structured, multi-level text as an intermediate target acts as a powerful semantic regularizer that aligns perfectly with how the human brain abstracts visual experiences.
  • The "Runaway Horse" Fix: Forcing an MLLM to generate a highly constrained, short summary before generating detailed descriptions prevents semantic drift and ensures the mined text remains grounded in the actual video stimulus.
  • Neuroscience Interpretability: By generating voxel-wise importance maps, the authors proved that their decoders actually look at the correct parts of the brain. For instance, the decoded motion narrative heavily relies on activations in the MT, MST, and TPOJ regions—the brain's well-established motion-processing hubs.

Key Results

SemVideo achieved state-of-the-art (SOTA) performance on 8 out of 10 evaluation metrics across both datasets, spanning semantic, pixel, and spatiotemporal levels.

  • Semantic Accuracy: On the CC2017 dataset, SemVideo achieved a SOTA 2-way video retrieval score of 0.865 and a 50-way video retrieval score of 0.264, drastically outperforming previous methods like Mind-Video and NeuroClips.
  • Temporal Coherence: It achieved the lowest Endpoint Error (4.788), which measures the mathematical distance between the optical flow of the original and reconstructed videos. Qualitatively, this means actions like "a person turning their head" are rendered smoothly without the subject morphing into a different object mid-motion.
Figure 1. Top: While a subject watches an original video stimulus, their brain activity is recorded via fMRI. Middle: Reconstructed results from previous methods, which suffer from two issues: Ap- pearance Mismatch and Motion Misalignment. Bottom: Recon- structed result from SemVideo, achieves both semantic consis- tency (reconstructing the “kitten”) and motion coherence (match- ing dynamic actions like “crouching” and “turning”) by leveraging hierarchical semantic descriptions as intermediate targets to guide the fMRI signal decoding process.

Limitations and Future Work

The authors noted a distinct failure case regarding abrupt boundary transitions. Because the experimental datasets are created by evenly segmenting long movies into short clips, some clips start or end with sudden, artificial cuts. Because SemVideo's motion decoder is designed to optimize for smooth, continuous temporal transitions, it struggles to replicate these sudden, unnatural scene shifts.

Future work will likely focus on better handling of these boundary transitions and integrating even more advanced video generation backbones as the underlying technology matures.

Summary updated

Abstract

Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions. To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.
BibTeX
@inproceedings{yang2026semvideo,
  author = {Minghan Yang and Lan Yang and Ke Li and Honggang Zhang and Kaiyue Pang and Yizhe Song},
  title = {{SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=3FrJBnDfZx},
}
Figure