Introduction: The "BERTology" Problem
While transformer-based models like BERT, RoBERTa, and ALBERT demonstrate impressive performance on various benchmarks, the NLP community faces a "BERTology" problem: we lack a deep understanding of the specific linguistic rules these models have actually mastered. A common evaluation method is "probing," where a simple classifier is trained on a model's frozen representations to detect linguistic features. However, high probing scores can be deceptive, as models might rely on simple heuristics or "shortcuts" rather than genuine structural knowledge.
This study investigates this phenomenon by focusing on Relative Clauses (RCs) in American English. RCs serve as an ideal stress test because they require models to link a relativizer (who, which, that) to an antecedent while respecting both complex grammatical rules and semantic constraints like animacy.
Methodology
The researchers employed a three-pronged strategy to evaluate linguistic depth:
- Probing Classifiers: Logistic regression classifiers were trained on sentence embeddings to distinguish between grammatical and ungrammatical RCs using a naturalistic dataset of ~48,000 sentences from the Corpus of Contemporary American English (COCA).
- Diagnostic Cases: A targeted "stress test" set of 32 sentences evaluated challenges such as long-distance dependencies and "agreement attractors" (distractor nouns that might trigger incorrect relativizer choices).
- Masked Language Modeling (MLM) Task: The models' native "fill-in-the-blank" capabilities were tested by masking relativizers or antecedents to see if predictions remained grammatically and semantically plausible.
Results: The Probing Illusion
All three models achieved high probing accuracy, ranging from 83.0% to 85.5%. However, non-contextualized baselines like GloVe also performed surprisingly well, suggesting that the models may be exploiting simple surface-level patterns (e.g., the mere presence of the word who) rather than deep syntactic structures.
As shown in the table above, while the transformer models outperform baselines on the total score, specific modifications like "which → who" reveal significant drops in performance for some models, indicating a fragile grasp of certain semantic constraints.
Layer-wise Analysis and Model "Personalities"
Analysis of the models' internal layers shows that linguistic knowledge for RCs emerges very early, typically between layers 1 and 5, before plateauing in the higher layers.
The study also identified distinct "personalities" among the models:
- RoBERTa acts as a "Semantic Specialist," showing high confidence in detecting agreement errors but extreme sensitivity to sentence length and distance.
- BERT and ALBERT are more "Structuralist," handling long-distance dependencies better than RoBERTa but proving more susceptible to confusion from agreement attractors.
Conclusion
The findings suggest that models are generally better at following strict grammatical rules (like subject-verb agreement) than nuanced semantic ones (like animacy). Probing alone tends to overestimate true linguistic "understanding." Future evaluations should combine probing with diagnostic tests and native task evaluations to provide a more accurate picture of a model's capabilities. Expanding this research to larger model scales and non-English languages remains a critical next step.
Summary updated