Stabilizing Neural-to-Image Retrieval with Frequency-Domain Priors and Adaptive Supervision

YaoYang Liu, ZhuoShi Tang, Haopeng Lai, Weijun Sun, Xiaozhao Fang, Jie WenarXiv 2026 · arXiv

TL;DR

This paper introduces a novel framework to improve how we "read" visual thoughts from brain activity (EEG/MEG). By incorporating frequency-domain statistics and adaptively adjusting the strength of supervision based on signal stability, the authors achieved state-of-the-art results in mapping neural signals to a shared semantic space with images, significantly outperforming existing benchmarks in both individual and cross-subject settings.

What Problem Does This Paper Address?

Visual neural decoding—the process of inferring what someone is looking at based on their brain activity—is a holy grail for brain-computer interfaces (BCIs). While we've made huge strides by using pretrained vision-language models like CLIP to provide a "semantic map," a massive bottleneck remains: alignment.

Brain signals (EEG and MEG) are notoriously "messy." They are noisy, vary wildly from one trial to the next, and look different from person to person. Most current methods treat every brain recording with the same level of "trust" during training. This paper argues that this "one-size-fits-all" approach fails because it doesn't account for the inherent instability and heterogeneity of neural data.

Methods

The authors propose a unified framework designed to stabilize this alignment process through three key innovations:

  1. Frequency-Domain Calibration: Instead of just looking at raw time-domain waveforms, the model extracts amplitude and phase statistics using a Fast Fourier Transform (FFT). These features act as "priors" that help recalibrate the neural embeddings via Feature-wise Linear Modulation (FiLM), ensuring the model captures rhythmic brain patterns that are often invisible in raw data.
  2. Phase-Guided Adaptive Supervision: Not all brain responses are equally reliable. The framework calculates "phase-circular variance"—a measure of how synchronized and stable the neural response is. This value is used to dynamically adjust the "temperature" in the contrastive loss function. Essentially, the model "listens" more closely to stable, high-quality signals and relaxes its supervision for noisier ones.
  3. Dual-Stream Low-Dimensional Adaptation: To prevent the model from learning redundant noise, the authors project both brain and image features into a compact, low-dimensional shared subspace. They use a specific correlation regularization (inspired by Barlow Twins) to ensure the dimensions are informative and non-redundant.
Fig. 1. Overview of the proposed neural-to-image retrieval framework. Left: A frozen CLIP visual encoder first defines the target image semantic space, and the brain encoder maps neural signals into this shared space. To improve the stability of cross-modal alignment, the framework combines dual-stream adaptation, batch-wise standardization, and low-dimensional shared-subspace regularization. In parallel, a frequency-domain branch extracts amplitude and phase statistics from neural signals to conditionally calibrate neural embeddings. Phase-circular variance is further used to derive a phase-stability-based control signal, which generates sample-specific temperatures for adaptive contrastive learning. By jointly integrating frequency-based calibration, phase-based supervision control, and cross-modal structural regularization, the framework improves the robustness and retrieval performance of neural-to-image alignment. Right: Internal structure of the low-dimensional adapter. The adapter takes the input embedding, applies normalization and two lightweight linear transformations, and produces both the refined output and its low-dimensional bottleneck representation.

Datasets

The framework was evaluated on two large-scale, publicly available benchmarks:

  • THINGS-EEG: Recordings from 10 subjects viewing 1,654 object concepts.
  • THINGS-MEG: High-temporal resolution recordings from 4 participants viewing 1,854 concepts.

These datasets are particularly challenging because they move beyond simple "closed-set" classification (e.g., "is this a cat or a dog?") to "open-vocabulary" retrieval among hundreds of possible categories.

Key Insights and Ideas

  • Phase as a Proxy for Reliability: One of the most elegant "aha" moments is using circular phase statistics to modulate training. If the phase of the brain signal is consistent, the signal is likely a "clean" representation of the stimulus, justifying a sharper training objective.
  • Stability Over Complexity: The authors found that simply making the neural encoder "deeper" or more complex doesn't help if the alignment pathway is unstable. Stabilization techniques (like batch-wise standardization and low-dim bottlenecks) are more critical than raw parameter count.
  • Synergy of Priors: Frequency-domain features and adaptive supervision work better together than alone. The frequency priors provide the "what" (better features), while adaptive supervision provides the "how much" (how much to trust those features).

Key Results

The proposed method set new performance bars for the 200-way zero-shot retrieval task:

  • THINGS-EEG (Intra-subject): Achieved 63.40% Top-1 accuracy, a massive 12.5% improvement over the previous state-of-the-art (UBP).
  • THINGS-EEG (Inter-subject): In the much harder task of decoding a new subject the model hasn't seen before, it reached 24.05% Top-1 accuracy, outperforming baselines by over 11%.
  • Qualitative Success: As seen in the retrieval examples, the model often retrieves the exact ground-truth image or semantically very similar objects (e.g., retrieving different types of animals or tools when the exact match is missed).
Fig. 2. Qualitative comparison of neural-to-image retrieval results. The first column shows the ground-truth target image (GT), and the remaining columns show the Top-1 to Top-5 retrieved results for each method. Red boxes denote correct matches.

Limitations and Future Work

While impressive, the authors acknowledge several hurdles:

  • Descriptor Simplicity: The current frequency branch uses relatively compact statistical descriptors. Future work could involve more complex time-frequency architectures (like wavelets or spectrogram-based transformers) to capture finer neural dynamics.
  • Neuroscientific Validation: While the "control signal" improves engineering performance, the exact neuroscientific reason why certain frequency enhancements help more for specific samples needs deeper investigation.
  • Fine-Grained Challenges: The model still struggles with fine-grained discrimination between very similar categories (e.g., different breeds of dogs), suggesting that the "semantic gap" between noisy EEG and high-res visual embeddings isn't fully closed yet.

Summary updated

Abstract

Visual neural decoding aims to infer perceived visual stimuli from brain activity, offering important insights into human visual cognition while enabling transformative applications in brain computer interfaces and artificial intelligence. Recent advances in pretrained vision-language models have provided a semantically rich representation space for this task. However, reliable alignment between EEG/MEG signals and visual embeddings remains challenging because neural responses are noisy, highly variable across trials, and subject dependent. To address this issue, we propose a unified neural-to-image retrieval framework that improves retrieval performance by aligning neural and image representations in a shared semantic space. The framework integrates dual-stream adaptation, frequency-domain calibration, and phase-guided temperature modulation to promote more robust cross-modal semantic alignment. Specifically, dual-stream adaptation models a low-dimensional shared subspace with correlation regularization to suppress redundant noise and enhance cross-modal consistency. Frequency-domain calibration conditionally modulates neural representations using amplitude and phase statistics. Phase-guided temperature modulation further adjusts the supervision strength in contrastive learning according to the stability of neural responses. Experiments on THINGS-EEG and THINGS-MEG under both intra-subject and inter-subject settings demonstrate that the proposed framework consistently outperforms strong baselines. These results suggest that robust neural-to-image retrieval under heterogeneous neural conditions benefits from a stable shared semantic space, frequency-informed representation calibration, and adaptive supervision. Code will be released at github.com/wanfkuabc/Geometry-Regularized-Brain-Embeddings.
BibTeX
@misc{liu2026stabilizing,
  author = {YaoYang Liu and ZhuoShi Tang and Haopeng Lai and Weijun Sun and Xiaozhao Fang and Jie Wen},
  title = {{Stabilizing Neural-to-Image Retrieval with Frequency-Domain Priors and Adaptive Supervision}},
  year = {2026},
  eprint = {2611.24163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24163},
}
Figure