TL;DR
This paper argues that current "syntactic probing" methods, used to assess if language models like BERT understand syntax, are flawed because they rely on semantic cues in the training data. By evaluating probes on syntactically correct but semantically nonsensical "Jabberwocky" sentences, the authors show a significant drop in performance, suggesting that models might be exploiting semantic shortcuts rather than truly learning syntax.
What Problem Does This Paper Address?
A major area of research in NLP is understanding whether large, pre-trained language models (like BERT, GPT-2, RoBERTa) implicitly learn linguistic structures, particularly syntax. A common method to investigate this is "probing." In this approach, a small, supervised model (the "probe") is trained to extract a specific linguistic feature (e.g., dependency relations) from the internal representations of a larger language model. If the probe performs well, it's often concluded that the language model has encoded that linguistic feature.
However, this paper questions the validity of these conclusions. The authors argue that existing probing studies often fail to isolate syntax from semantics. This means that probes might be learning to exploit statistical correlations between words that happen to co-occur in meaningful sentences, rather than genuinely understanding the underlying grammatical structure. This is problematic because it could lead to overestimating the syntactic capabilities of language models. The core problem is the potential conflation of syntax and semantics in the evaluation of these probes.
Methods
The authors introduce a novel approach to disentangle syntax from semantics in probing. Their key method involves creating a new evaluation dataset and using it to test existing probing techniques.
- Jabberwocky Corpus Generation: They construct a corpus of English sentences that are syntactically well-formed but semantically nonsensical. This is inspired by Lewis Carroll's "Jabberwocky" poem and Noam Chomsky's famous example "Colorless green ideas sleep furiously." They achieve this by taking existing syntactically annotated sentences (from the English EWT Universal Dependency treebank) and replacing real words with pseudowords that match the part-of-speech tags and morphological features of the original words. This ensures the sentences have valid syntactic structures but lack inherent meaning.
- Probe Evaluation: They then take two established syntactic probes (a "structural probe" and a "perceptron probe," both designed to predict syntactic distance between words) and train them on standard, semantically meaningful data. Crucially, they then evaluate these already trained probes on their newly generated Jabberwocky corpus. The hypothesis is that if the probes are truly learning syntax, their performance should not degrade significantly when the semantic content is removed.
Datasets
- Training Data: Standard English Universal Dependency (UD) treebanks were used to train the probes.
- Evaluation Data:
- Unchanged Sentences: Standard test sets from UD treebanks were used as a baseline for evaluation.
- Jabberwocky Sentences: A novel corpus of syntactically valid, semantically nonsensical sentences generated by the authors using pseudowords from the ARC Nonword Database.
Key Insights and Ideas
- Syntactic Probes Can Be Deceived by Semantics: The most significant insight is that current syntactic probes often rely on semantic cues present in natural language training data. When these cues are removed (by using pseudowords), the probes' performance drops considerably.
- Overestimation of Model Syntax Knowledge: The substantial performance degradation suggests that previous claims about language models' syntactic understanding, based on these probes, might be inflated. Models might be learning statistical word co-occurrence patterns rather than abstract syntactic rules.
- The Need for Semantically Neutral Evaluation: The paper strongly advocates for the necessity of evaluating syntactic understanding using data that is devoid of semantic meaning. This is crucial for accurately assessing a model's true syntactic capabilities.
- Jabberwocky as a Diagnostic Tool: The generated Jabberwocky corpus serves as a powerful diagnostic tool to expose the limitations of current probing methodologies.
Key Results
- Performance Drop on Jabberwocky: When evaluated on the Jabberwocky sentences, the probes trained on BERT, GPT-2, and RoBERTa showed a significant decrease in performance compared to their performance on normal sentences.
- Perceptron Probe Impact: For one probe (the perceptron probe), the average UUAS (Unlabeled Undirected Attachment Score) for BERToids dropped by 15.4 points.
- Reduced Advantage Over Baselines: While the models still generally outperformed simple baselines (like FastText embeddings or random guessing), their lead was substantially reduced. For instance, BERT's advantage over a Fast+Pos baseline was reduced by 53% for one probe.
- GPT-2 Struggles: In some cases, GPT-2 even performed worse than simple baselines on the Jabberwocky data.
Limitations and Future Work
- Limitations of Current Probes: The authors acknowledge that the observed performance drop might also reflect a weakness in the probes themselves, not just the models. They suggest that future work should train probes on Jabberwocky data to better understand their capabilities.
- Need for More Nuanced Experimentation: The paper calls for more sophisticated evaluation methods than simply training a syntactic probe as if it were a parser.
- Separation of Syntax and Semantics: The core recommendation for future work is to explicitly separate syntax and semantics in probing experiments.
- Developing Artificial Treebanks: The authors suggest exploring the development of artificial treebanks specifically designed for training syntactic probes, which would minimize confounding statistical biases.
- Code and Data Availability: The authors have made their Jabberwocky evaluation data and code publicly available, encouraging further research in this direction.
Summary updated