TL;DR
The authors introduce EEGProbe, a new benchmark designed to test whether general-purpose Vision-Language Models (VLMs) can actually interpret raw, minimally processed EEG signals. Their evaluation of 11 top-tier models reveals a significant gap: even the best model (GPT-5.6 Sol) scores only 41.63%, struggling particularly with matching brain activity to external stimuli like images or audio.
What Problem Does This Paper Address?
We know that VLMs like GPT-4 or Gemini are incredibly capable at describing photos or reasoning through text. But can they "read" the electrical language of the human brain?
Electroencephalography (EEG) data is notoriously difficult to work with—it is noisy, non-stationary, and varies wildly between different people and recording setups. Traditionally, decoding EEG requires highly specialized "black box" models or heavy feature engineering (like Fourier transforms).
This paper asks a fundamental question for neural engineering: Can a general-purpose multimodal model extract meaningful information from raw EEG signals without being specifically trained for it? To answer this, the researchers needed a way to evaluate these models that prevents them from "cheating" using text-based shortcuts or prior knowledge.
Methods
The researchers developed a rigorous pipeline to transform raw EEG data into a format VLMs can digest while ensuring the evaluation is grounded strictly in the signal itself.
- Serialization: Multichannel EEG arrays are converted into structured numerical text (e.g., "Channel Fz: 2.9, -5.0..."). This preserves the temporal structure without adding handcrafted features.
- Benchmark Construction: The team curated 620 question-answer pairs across 13 subtasks. These are grouped into three categories:
- Signal Classification: Identifying states like sleep stages or seizures.
- Cross-modal Alignment: Matching an EEG segment to a corresponding image, text, audio, or video.
- Neurophysiological Reasoning: Explaining why a signal looks a certain way (e.g., frequency-band analysis).
- The Signal-Necessity Test: To ensure models aren't just good guessers, every question is first given to a panel of LLMs without the EEG data. If a model can answer correctly based only on the text of the question, that question is discarded from the benchmark.
- Expert Verification: Every remaining instance is manually reviewed by EEG experts to ensure the ground truth is correct and the provided signal actually contains enough evidence to support the answer.
Datasets
EEGProbe curates data from 15 publicly available datasets, covering a vast range of experimental paradigms:
- Classification: BCI Competition IV 2a (motor imagery), SEED/SEED-IV (emotion), Sleep-EDFx (sleep staging), and CHB-MIT (seizure detection).
- Alignment: SparrKULee (audio), THINGS-EEG2 (images), and ZuCo 1.0 (natural language reading).
- Reasoning: ERP CORE, TUAR v3.0.1, and the CAP Sleep Database.
Key Insights and Ideas
- The Alignment Bottleneck: Models performed worst on cross-modal alignment (matching EEG to an image or sound), often hovering near the 25% chance level. This suggests that while VLMs can "read" the numbers, they can't yet map those numbers to the semantic concepts they represent in other modalities.
- Reasoning > Classification: Interestingly, models were often better at reasoning about signal properties (like identifying a late positive deflection) than they were at simple classification. This suggests they have the "book knowledge" of neurophysiology but struggle to apply it to noisy real-world labels.
- Guidance is the Key: The researchers found that providing "neurophysiological guidance"—essentially telling the model what specific patterns or brain regions to look for—significantly boosted performance. For example, in EEG-image matching, guidance improved accuracy from 20% to 40%. This implies the information is in the signal, but the models need a "nudge" to focus on the right features.
- Scale Isn't Everything: Performance did not strictly correlate with model size. Some smaller, locally-run models outperformed larger ones, indicating that how a model is tuned for multimodal reasoning matters more than raw parameter count.
Key Results
The evaluation of 11 representative VLMs (including GPT-5.6 Sol, Gemini-3.8-Flash, and DeepSeek-V4.1-Flash) under a zero-shot protocol yielded the following:
- Overall Performance: The top model, GPT-5.6 Sol, achieved a total score of 41.63%. The average across all models was a mere 33.88%.
- Classification: Models showed some ability to detect seizures (~53% avg) and ERPs (~50% avg), but struggled with motor imagery (~26% avg).
- Alignment: This was the major failure point, with average scores across all four sub-modalities (Image, Text, Audio, Video) staying between 23% and 27%—barely above random guessing.
- Reasoning: This category showed the most promise and the highest variance, with some models scoring as high as 80% on frequency-band analysis while others failed completely.
Limitations and Future Work
The authors acknowledge that numerical serialization (turning signals into lists of numbers) might be a bottleneck. VLMs are primarily trained on pixels and text tokens; they may not be optimized to perceive high-frequency oscillations represented as strings of digits.
Future work could involve:
- Developing EEG-specific tokenizers that allow VLMs to "see" the signal more naturally.
- Moving beyond zero-shot evaluation to see how minimal fine-tuning affects understanding.
- Expanding the benchmark to include more complex, multi-step clinical reasoning tasks.
EEGProbe serves as a sobering reminder that while AI is advancing rapidly, truly "grounding" these models in complex physiological signals remains a significant frontier.
Summary updated