SIRAZ: Sparse Inter-Layer Residual Aggregation for Zero-Shot Visual Neural Decoding

Neha, Mudasir GanaiearXiv 2026 · arXiv

TL;DR

The SIRAZ framework significantly advances visual neural decoding by aligning brain signals (EEG/MEG) with biologically inspired, hierarchical image representations. By combining retina-inspired visual transformations with a novel multi-scale temporal encoder for neural dynamics, the authors achieve state-of-the-art zero-shot image retrieval, reaching 92.5% top-1 accuracy on the THINGS-EEG dataset.

What Problem Does This Paper Address?

Visual neural decoding aims to "read" what a person is seeing by analyzing their brain activity. While promising for brain-computer interfaces (BCIs), this task faces two major hurdles:

  1. The Representational Gap: Standard computer vision models process images differently than the human visual system. Most existing decoding methods rely on the final, highly abstract layers of pretrained models, ignoring the hierarchical, multi-stage nature of biological vision.
  2. Signal Complexity: Neural signals like Electroencephalography (EEG) and Magnetoencephalography (MEG) have a very low signal-to-noise ratio and exhibit high variability across different subjects and time scales.

The authors tackle these by asking: Can we improve decoding by making our artificial vision pipeline look more like a human one?

Methods

The proposed SIRAZ (Sparse Inter-layer Residual Aggregation for Zero-shot) framework consists of three innovative components designed to bridge the gap between machine and biological vision:

  • Biologically Inspired Visual Streams: Instead of feeding raw images into a model, SIRAZ creates two parallel streams:
  • Foveated Stream: Mimics the human eye by preserving high resolution at the center (fixation point) and blurring the periphery.
  • Magnocellular-inspired Stream: Uses "Difference-of-Gaussians" (DoG) transformations to emphasize luminance transitions and spatial contrast, mimicking the brain's early motion and edge detection pathways.
  • Residual Gated Refinement Adapter (ReGRAM): Rather than just using the final output of a vision backbone (like DINOv2), SIRAZ extracts features from multiple intermediate layers. These are aggregated and refined through a gated adapter mechanism that learns which "depths" of the vision model best correspond to neural activity.
  • Direction-aware Multi-Scale Temporal Attention Encoder (DMSTAE): On the neural side, this encoder processes EEG/MEG signals by looking at the direction of signal changes (positive vs. negative transitions) across multiple temporal resolutions. This helps the model capture the fast, complex dynamics of visual processing in the brain.
Figure 1: Overview of the proposed SIRAZ architecture. (i) Direction-aware Multi-Scale Temporal Attention Encoder for M/EEG representation learning, (ii) hierarchical intermediate-layer representation learning from a pretrained vision model, and (iii)Residual Gated Refinement Adapter Module (ReGRAM).

Datasets

The framework was evaluated on two large-scale public benchmarks:

  • THINGS-EEG: 63-channel EEG data from 10 participants viewing 1,854 object concepts.
  • THINGS-MEG: High-density 271-channel MEG data from 4 participants.

Both datasets use a zero-shot protocol, meaning the model is tested on its ability to retrieve images belonging to categories it never saw during the training phase.

Key Insights and Ideas

  • Intermediate Layers are Essential: The paper demonstrates that the brain's neural responses align better with mid-level visual features than with the final, purely semantic outputs of vision transformers. Adaptive selection of these layers is a key driver of performance.
  • Directionality Matters in Neural Signals: Standard convolutions often ignore whether a signal is increasing or decreasing. By explicitly modeling the direction of temporal transitions, the DMSTAE captures more discriminative "neural signatures."
  • Complementary Visual Pathways: The foveated and magnocellular streams capture different, non-redundant information. Combining them provides a much richer target for the neural signal to align with compared to using raw images alone.

Key Results

SIRAZ sets a new bar for zero-shot visual retrieval from brain signals:

  • THINGS-EEG (Intra-subject): Achieved 92.5% top-1 accuracy and 99.27% top-5 accuracy, vastly outperforming previous state-of-the-art methods like Shallow Align (82.6%) and BI-Cap (72.4%).
  • THINGS-MEG (Intra-subject): Achieved 50.25% top-1 accuracy, representing a significant jump over the previous best of 48.0%.
  • Inter-subject Generalization: While more challenging, SIRAZ also improved performance in "leave-one-subject-out" settings, reaching 22.31% top-1 accuracy on EEG, nearly double that of several recent baselines.
Table 1 Top-1 and Top-5 retrieval accuracy (%) for 200-way zero-shot retrieval on the THINGS-EEG dataset.

Limitations and Future Work

The primary limitation identified is the performance gap between intra-subject and inter-subject decoding. While SIRAZ performs exceptionally well when trained and tested on the same person, generalizing those learned representations to a completely new brain remains difficult due to inherent biological variability.

The authors suggest that future work should focus on developing more robust "foundation models" for neural data that can better handle this cross-subject variance, potentially through larger-scale pretraining or more advanced domain adaptation techniques.

Summary updated

Abstract

Brain-computer interfaces aim to establish direct communication between neural activity and external systems, making interpretation of brain signals an important problem. In this context, neural visual decoding seeks to infer visual information from neural activity by learning a mapping between brain responses and the corresponding visual stimuli. It investigates how visual information emerges from neural signals and relates to perceptual processing while enabling the reconstruction or retrieval of visual stimuli from neural signals. Our brain primarily follows a hierarchical pathway to process visual information captured by the eye. However, neural representations learned by existing methods differ fundamentally from those produced by the biological visual system. To bridge this gap and mimic the human vision hierarchy, we introduce Sparse Inter-layer Residual Aggregation for Zero-shot (SIRAZ). SIRAZ is a framework that aligns neural visual representations by jointly modeling early visual processes via fovea- and magnocellular-pathway-inspired Difference-of-Gaussians transformations. To align the semantic information with neural information, it extracts multiple intermediate representations of the image at multiple depths and passes them through the Residual Gated Adapter Mechanism. To study the temporal dynamics of neural signals, the Direction-aware Multi-Scale Temporal Attention Encoder is introduced to align the learned neural representation with the final visual representation. The proposed framework achieves 92.5% top-1 and 99.27% top-5 accuracy in the intra-subject setting on the THINGS-EEG dataset, and 50.25% and 75% top-1 and top-5 accuracy in the THINGS-MEG dataset. The results indicate that combining biologically motivated visual representations with the structured modalities of neural dynamics improves cross-modal alignment and zero-shot image retrieval.
BibTeX
@misc{neha2026siraz,
  author = {Neha and Mudasir Ganaie},
  title = {{SIRAZ: Sparse Inter-Layer Residual Aggregation for Zero-Shot Visual Neural Decoding}},
  year = {2026},
  eprint = {2611.23991},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23991},
}
Figure