EEGProbe: Do Vision-Language Models Really Understand EEG? A Benchmark for Signal-Grounded EEG Signals Evaluation

Haozhe Zhu, Dan Yuan, Chenghao Gu, Feike Xu, Yue Wen, Shin'ichi KonomiarXiv 2026 · arXiv

TL;DR

The authors introduce EEGProbe, a new benchmark designed to test whether general-purpose Vision-Language Models (VLMs) can actually interpret raw, minimally processed EEG signals. Their evaluation of 11 top-tier models reveals a significant gap: even the best model (GPT-5.6 Sol) scores only 41.63%, struggling particularly with matching brain activity to external stimuli like images or audio.

What Problem Does This Paper Address?

We know that VLMs like GPT-4 or Gemini are incredibly capable at describing photos or reasoning through text. But can they "read" the electrical language of the human brain?

Electroencephalography (EEG) data is notoriously difficult to work with—it is noisy, non-stationary, and varies wildly between different people and recording setups. Traditionally, decoding EEG requires highly specialized "black box" models or heavy feature engineering (like Fourier transforms).

This paper asks a fundamental question for neural engineering: Can a general-purpose multimodal model extract meaningful information from raw EEG signals without being specifically trained for it? To answer this, the researchers needed a way to evaluate these models that prevents them from "cheating" using text-based shortcuts or prior knowledge.

Methods

The researchers developed a rigorous pipeline to transform raw EEG data into a format VLMs can digest while ensuring the evaluation is grounded strictly in the signal itself.

  1. Serialization: Multichannel EEG arrays are converted into structured numerical text (e.g., "Channel Fz: 2.9, -5.0..."). This preserves the temporal structure without adding handcrafted features.
  2. Benchmark Construction: The team curated 620 question-answer pairs across 13 subtasks. These are grouped into three categories:
  3. Signal Classification: Identifying states like sleep stages or seizures.
  4. Cross-modal Alignment: Matching an EEG segment to a corresponding image, text, audio, or video.
  5. Neurophysiological Reasoning: Explaining why a signal looks a certain way (e.g., frequency-band analysis).
  6. The Signal-Necessity Test: To ensure models aren't just good guessers, every question is first given to a panel of LLMs without the EEG data. If a model can answer correctly based only on the text of the question, that question is discarded from the benchmark.
  7. Expert Verification: Every remaining instance is manually reviewed by EEG experts to ensure the ground truth is correct and the provided signal actually contains enough evidence to support the answer.
Overview of the EEGProbe construction pipeline, highlighting the signal-necessity test and expert verification phases.

Datasets

EEGProbe curates data from 15 publicly available datasets, covering a vast range of experimental paradigms:

  • Classification: BCI Competition IV 2a (motor imagery), SEED/SEED-IV (emotion), Sleep-EDFx (sleep staging), and CHB-MIT (seizure detection).
  • Alignment: SparrKULee (audio), THINGS-EEG2 (images), and ZuCo 1.0 (natural language reading).
  • Reasoning: ERP CORE, TUAR v3.0.1, and the CAP Sleep Database.

Key Insights and Ideas

  • The Alignment Bottleneck: Models performed worst on cross-modal alignment (matching EEG to an image or sound), often hovering near the 25% chance level. This suggests that while VLMs can "read" the numbers, they can't yet map those numbers to the semantic concepts they represent in other modalities.
  • Reasoning > Classification: Interestingly, models were often better at reasoning about signal properties (like identifying a late positive deflection) than they were at simple classification. This suggests they have the "book knowledge" of neurophysiology but struggle to apply it to noisy real-world labels.
  • Guidance is the Key: The researchers found that providing "neurophysiological guidance"—essentially telling the model what specific patterns or brain regions to look for—significantly boosted performance. For example, in EEG-image matching, guidance improved accuracy from 20% to 40%. This implies the information is in the signal, but the models need a "nudge" to focus on the right features.
  • Scale Isn't Everything: Performance did not strictly correlate with model size. Some smaller, locally-run models outperformed larger ones, indicating that how a model is tuned for multimodal reasoning matters more than raw parameter count.
Representative task examples from the three categories of EEGProbe.

Key Results

The evaluation of 11 representative VLMs (including GPT-5.6 Sol, Gemini-3.8-Flash, and DeepSeek-V4.1-Flash) under a zero-shot protocol yielded the following:

  • Overall Performance: The top model, GPT-5.6 Sol, achieved a total score of 41.63%. The average across all models was a mere 33.88%.
  • Classification: Models showed some ability to detect seizures (~53% avg) and ERPs (~50% avg), but struggled with motor imagery (~26% avg).
  • Alignment: This was the major failure point, with average scores across all four sub-modalities (Image, Text, Audio, Video) staying between 23% and 27%—barely above random guessing.
  • Reasoning: This category showed the most promise and the highest variance, with some models scoring as high as 80% on frequency-band analysis while others failed completely.

Limitations and Future Work

The authors acknowledge that numerical serialization (turning signals into lists of numbers) might be a bottleneck. VLMs are primarily trained on pixels and text tokens; they may not be optimized to perceive high-frequency oscillations represented as strings of digits.

Future work could involve:

  • Developing EEG-specific tokenizers that allow VLMs to "see" the signal more naturally.
  • Moving beyond zero-shot evaluation to see how minimal fine-tuning affects understanding.
  • Expanding the benchmark to include more complex, multi-step clinical reasoning tasks.

EEGProbe serves as a sobering reminder that while AI is advancing rapidly, truly "grounding" these models in complex physiological signals remains a significant frontier.

Summary updated

Abstract

Vision-language models (VLMs) have shown strong multimodal capabilities across scientific and biomedical domains, yet their ability to understand electroencephalography (EEG) signals remains unclear. We introduce EEGProbe, a signal-grounded benchmark for evaluating whether general-purpose VLMs can extract and reason from minimally processed, channel-wise EEG without task-specific feature engineering or EEG-specific adaptation. EEGProbe contains 620 EEG-centered questions across 13 subtasks and three capability categories: EEG Signal Classification, EEG Cross-modal Alignment, and EEG Neurophysiological Reasoning. To reduce shortcut-based answering, EEGProbe preserves the temporal structure of the recordings where possible, minimizes benchmark-induced processing, and applies signal-necessity testing followed by expert verification. We evaluate 11 representative open- and closed-source VLMs under a unified zero-shot protocol. The strongest model, GPT-5.6 Sol, achieves an overall score of only 41.63\%, with cross-modal EEG alignment emerging as a major bottleneck. We further examine model behavior from a neurophysiological perspective and analyze how temporal, spectral, spatial, and physiological EEG structure relates to model successes and failures. Inference-time neurophysiological guidance can reshape model predictions by helping models recognize the neurophysiological patterns present in the input and connect them with relevant neurological knowledge, narrowing the gap between general multimodal competence and EEG understanding. EEGProbe provides a testbed for current VLMs and a basis for developing future multimodal models that can interpret EEG evidence more reliably. The EEGProbe benchmark will be publicly available at https://huggingface.co/datasets/2122ZHU/EEGProbe.
BibTeX
@misc{zhu2026eegprobe,
  author = {Haozhe Zhu and Dan Yuan and Chenghao Gu and Feike Xu and Yue Wen and Shin'ichi Konomi},
  title = {{EEGProbe: Do Vision-Language Models Really Understand EEG? A Benchmark for Signal-Grounded EEG Signals Evaluation}},
  year = {2026},
  eprint = {2611.20779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20779},
}
Figure