Abstract: Do large language models (LLMs) like BERT truly "know" grammar, or are they just really good at using context clues? This paper demonstrates that the "syntactic probes" used to claim LLMs understand syntax are heavily reliant on semantic shortcuts. When tested on "Jabberwocky" sentences—which are grammatically perfect but semantically nonsensical—the performance of these probes plummets, suggesting we have yet to truly isolate pure syntactic knowledge in neural networks.
Introduction
In the NLP community, it is a common claim that models like BERT implicitly learn the rules of grammar (syntax) during their unsupervised pre-training. To prove this, researchers use probing: they train a small, supervised model (a "probe") to extract syntactic structures, like dependency trees, from the LLM's internal representations. If the probe succeeds, we conclude the LLM "knows" syntax.
However, the authors argue this logic is flawed due to the conflation of syntax and semantics. In natural language, "The chef cooked the meal" follows clear grammatical rules, but it also makes sense semantically. A probe might achieve high accuracy not because it found a "syntax neuron," but because it recognized that "chef" and "cooked" often appear in specific structural relationships. To truly claim a model knows syntax, it must be able to process structure independently of meaning—much like Noam Chomsky’s famous example: "Colorless green ideas sleep furiously."
Methodology: Jabberwocky Probing
The researchers designed a clever "stress test" for syntactic probes using Jabberwocky probing. They took existing sentences from the English EWT (Universal Dependencies) treebank and replaced the content words with "pseudowords" from the ARC Nonword Database. These nonsense words follow English phonology and morphology (e.g., "The povicated briticists..."), preserving the exact syntactic structure while stripping away all real-world meaning.
They evaluated two popular types of probes—the Structural Probe (measuring word embedding distance) and the Perceptron Probe (a sequence-to-tree model)—across three "BERToids": BERT, GPT-2, and RoBERTa. Probes were trained on normal, meaningful English and then evaluated on both normal English and the new Jabberwocky sentences.
Key Results and Insights
The experiment revealed that probes are often "cheating" by relying on lexical and semantic cues. Because they perform significantly worse on Jabberwocky data than on normal data, it is clear they rely on semantics to "find" the syntax.
- Significant Performance Drop: For the Perceptron probe, the average performance across BERT-style models dropped by 15.4 UUAS points (Unlabeled Undirected Attachment Score) when moving from normal to Jabberwocky data.
- Shrinking Advantage: BERT’s lead over simple baselines (like a "Majority" baseline that predicts the most common tree shape) was reduced by 53% in some settings.
- GPT-2 Failure: In the structural probing task, GPT-2 actually performed worse than a simple "Path" baseline (which assumes every word just connects to its neighbor) when faced with Jabberwocky sentences.
Conclusion
This paper serves as an epistemological warning against over-interpreting probing results. Just because a probe can extract a tree doesn't mean the LLM is "using" that tree or that the representation is purely syntactic. While BERT generally maintained a higher lead over baselines than GPT-2—suggesting it may encode more robust structural information—current probes struggle to isolate it perfectly. If we want to claim a model "knows" syntax, we must move beyond simple accuracy scores and develop experiments that strictly separate the "how" (syntax) from the "what" (semantics).
Summary updated