Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning

Weihang You, Hanqi Jiang, Yi Pan, Junhao Chen, Tianming Liu, Fei DouarXiv 2026 · arXiv

TL;DR

NeuroAlign is a brain-inspired framework designed to bridge the gap between brain activity (fMRI) and visual stimuli (video). By mirroring the hierarchical and temporal processing of the human visual cortex through neural-temporal contrastive learning and discrete vector quantization, it achieves a massive 1.4–1.8× performance leap in cross-modal retrieval tasks over current state-of-the-art methods.

What Problem Does This Paper Address?

Decoding what a person sees based purely on their brain activity (fMRI) is a holy grail of cognitive neuroscience. However, aligning fMRI signals with video frames faces three major hurdles:

  1. The Hemodynamic Lag: Brain responses are temporally delayed and smeared by the Hemodynamic Response Function (HRF), lagging 6–10 seconds behind the actual visual stimulus.
  2. Low Signal-to-Noise Ratio (SNR): fMRI data is notoriously noisy, riddled with physiological artifacts.
  3. Hierarchical Processing: The human brain doesn't process vision all at once; it abstracts information sequentially across multiple cortical areas—from low-level edges (V1–V3) to complex objects (IT).

Existing methods often rely on continuous embeddings (like standard CLIP-style models), which are brittle under fMRI noise and fail to capture the brain's discrete, multi-scale hierarchical nature. NeuroAlign addresses these gaps by introducing a biologically plausible, discrete representation learning framework.

Methods

NeuroAlign implements a two-stage mechanism that closely mirrors biological visual pathways:

Overview of the NeuroAlign framework architecture, mapping the human visual processing system to the multi-modal alignment pipeline.
  • Neural-Temporal Contrastive Learning (NTCL): Instead of treating frames independently, NTCL models the temporal dynamics of both fMRI and video. It uses a learnable temporal adaptation layer to compensate for the 6–10 second HRF lag and employs a bidirectional transformer to predict future video features from past fMRI context using an InfoNCE loss.
  • Fine-Grained Pattern Matching (Vector Quantization): To combat low SNR, NeuroAlign uses a Vector Quantized Variational Autoencoder (VQ-VAE) to map continuous features into stable, discrete codes within a shared codebook. It handles the brain's hierarchical structure by aggregating information across multiple scales using dilated causal convolutions (TemporalAdapt) that mimic how different cortical areas integrate information over varying time scales.
  • DynaSyncMM-EMA: Standard codebook updates treat all modalities equally, which allows cleaner modalities (like video or text) to dominate the shared space. DynaSyncMM-EMA introduces a synchronized codebook update mechanism with variance-aware weighting. Higher-variance (higher-information) modalities receive higher weights, preventing noisy fMRI data from degrading the shared codebook.

Datasets

The authors validated their framework using two primary datasets:

  • Human Connectome Project (HCP): Utilized for fMRI pre-training, encompassing 1,200 subjects and 600,000 segments.
  • CC2017: Used for the cross-modal alignment and retrieval tasks. The training set consisted of 18 video clips (4,320 samples), and the test set contained 5 clips (1,200 samples). Video frames were temporally shifted by 6 seconds to align with the fMRI peak response.

Key Insights and Ideas

  • Discretization as a Noise Filter: Shifting from continuous embeddings to discrete vector quantization acts as a natural regularizer against physiological noise. By forcing continuous features to map to the nearest codebook entry, the model discards minor, noisy fluctuations.
  • True Component Synergy: The ablation studies revealed that no single component could compensate for another. Removing Pattern Matching caused a massive 60% performance collapse, while removing NTCL caused a 49% drop. The system relies entirely on the interplay between temporal modeling, noise-robust discretization, and balanced multi-modal fusion.
  • Variance-Aware Balancing: Treating fMRI and video equally during training introduces a sequential bias. Dynamically weighting updates based on modality variance ensures that the model learns a truly unified representational space without erasing modality-specific structures.

Key Results

NeuroAlign demonstrated substantial improvements across six bidirectional cross-modal retrieval tasks:

  • Massive Performance Leap: For fMRI-to-Video (F-V) retrieval, NeuroAlign achieved 50.31% Recall@5 and 64.64% Recall@10, representing a 1.8× improvement over the previous best method (NeuroClips) and a 10-fold improvement over standard tri-modal CLIP.
  • Unified Representation Space: The performance variance across all modality pairs (F-V, F-T, V-F) was less than 0.3%, proving that the model successfully mapped brain activity, video, and text into a highly synchronized, shared semantic space.
Qualitative visualization results showing that NeuroAlign successfully retrieves correct video frames and highlights localized regions with high activation correspondence to the query fMRI patterns.

Limitations and Future Work

While the paper demonstrates remarkable retrieval accuracy, certain limitations and future directions are noted:

  • Downstream Applications: The authors highlight potential future applications in clinical diagnostics, neuroprosthetics, and personalized neurotherapies.
  • Computational Constraints: Detailed discussion regarding computational scaling, real-time decoding constraints, or performance on larger, unaligned open-ended datasets was not explicitly discussed in the paper.

Summary updated

Abstract

Understanding neural responses to visual stimuli remains challenging due to the inherent complexity of brain representations and the modality gap between neural data and visual inputs. Existing methods, mainly based on reducing neural decoding to generation tasks or simple correlations, fail to reflect the hierarchical and temporal processes of visual processing in the brain. To address these limitations, we present NeuroAlign, a novel framework for fine-grained fMRI-video alignment inspired by the hierarchical organization of the human visual system. Our framework implements a two-stage mechanism that mirrors biological visual pathways: global semantic understanding through Neural-Temporal Contrastive Learning (NTCL) and fine-grained pattern matching through enhanced vector quantization. NTCL explicitly models temporal dynamics through bidirectional prediction between modalities, while our DynaSyncMM-EMA approach enables dynamic multi-modal fusion with adaptive weighting. Experiments demonstrate that NeuroAlign significantly outperforms existing methods in cross-modal retrieval tasks, establishing a new paradigm for understanding visual cognitive mechanisms.
BibTeX
@misc{you2026achieving,
  author = {Weihang You and Hanqi Jiang and Yi Pan and Junhao Chen and Tianming Liu and Fei Dou},
  title = {{Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning}},
  year = {2026},
  eprint = {2601.01339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.01339},
}
Figure