Abstract
Recent research has highlighted a surprising alignment between the internal states of deep learning language models and human brain activity. However, it remains unclear whether this correlation is driven by shared semantic comprehension or common structural processing. This study investigates this "syntax vs. semantics" dilemma by comparing a character-level LSTM's representations with human EEG data across three conditions: normal sentences, syntactically correct but nonsensical "Jabberwocky" sentences, and unstructured word-lists. Our findings reveal that the LSTM-brain correlation persists for Jabberwocky stimuli but fails for word-lists, suggesting that structural scaffolding, rather than meaning alone, is a key driver of model-brain alignment.
Introduction
The ability of Large Language Models (LLMs) and LSTMs to predict neural responses to language has sparked significant interest in computational neuroscience. While these models are often trained on massive corpora to predict the next token, their internal representations appear to mirror the hierarchical processing found in the human brain. A fundamental question persists: is this alignment due to a shared extraction of high-level meaning (semantics) or a shared adherence to structural rules (syntax)? By utilizing "out-of-distribution" stimuli like Jabberwocky—which preserves syntax while removing semantics—we can isolate the contribution of structural processing to this neural alignment.
Methods
The researchers developed a multi-stage pipeline to compare artificial and biological language processing.
Model and Data
A character-level LSTM was trained on one million Dutch Wikipedia sentences. The character-level architecture was essential for processing the pseudo-words found in Jabberwocky stimuli. Brain activity was captured via EEG from 27 native Dutch speakers listening to three types of stimuli:
- Sentence: Grammatically correct Dutch.
- Jabberwocky: Sentences with content words replaced by pseudo-words, preserving syntax and prosody.
- Word-list: Randomly shuffled words from the Sentence condition, removing all syntactic structure.
Decoding and Evaluation
A Ridge Regression model was trained to map EEG signals to the LSTM’s internal representations across various layers (Embedding, Convolutional, and LSTM layers 1-3). Performance was evaluated using a 2 vs. 2 classification test, determining if the model could correctly match EEG segments to their corresponding LSTM states.
Results
The study yielded several key insights into the nature of the brain-model interface.
The Primacy of Syntax
The decoding model successfully predicted LSTM states from EEG data for both normal sentences and Jabberwocky. For the Sentence condition, LSTM layers 1-3 achieved accuracies of 0.581, 0.600, and 0.569, respectively. Crucially, the Jabberwocky condition also showed significant decodability (0.565 for LSTM1 and 0.573 for LSTM2). In contrast, the Word-list condition failed to exceed chance levels, indicating that the alignment is dependent on contextual, structural processing rather than isolated word recognition.
The "Goldilocks" Layer
As shown in the results, performance was not uniform across the network. LSTM layer 2 consistently emerged as the "sweet spot," showing the strongest correlation with brain activity. Probing tasks further confirmed that this middle layer is highly specialized for capturing syntactic depth and structure before the model transitions to more task-specific output layers.
Discussion
These results demonstrate that the "brain-like" nature of LSTMs extends to the processing of nonsensical but structured language. The success of the Jabberwocky condition and the failure of the Word-list condition strongly suggest that syntax serves as the primary bridge for model-brain alignment.
While these findings are robust, limitations remain. The potential influence of positional information (word order) could not be entirely decoupled from syntactic processing. Furthermore, the modality gap between text-trained models and audio-based EEG recordings suggests that future work should explore audio-native architectures. Understanding the specific neural mechanisms—such as delta or theta oscillations—that align with model gating activities remains a promising frontier for future research.
Summary updated