Abstract
While Masked Language Models (MLMs) achieve high scores on standard linguistic probing tasks, they often fail on fine-grained diagnostic tests and complex semantic constraints. This research investigates whether the high performance of models like BERT reflects a deep understanding of grammar and semantics or "short-cut" learning. By focusing on Relative Clauses (RCs) in American English, the authors demonstrate that high probing accuracy can mask significant model-specific weaknesses, particularly in handling nuanced grammatical rules and animacy constraints.
Introduction: Beyond Probing Accuracy
The "BERTology" era has established transformer-based models as incredibly capable, yet the nature of the "knowledge" they acquire remains opaque. A common evaluation method is probing, where a simple classifier is trained on a model's frozen internal representations to detect grammatical properties. However, the authors argue that probing alone is insufficient to capture the depth of linguistic understanding.
They focus on Relative Clauses (RCs) (e.g., "The student who studied passed"), which are linguistically complex. RCs require a model to link a "relativizer" (who, which, that) to an "antecedent" (student), necessitating an understanding of both structural grammar and semantic constraints like animacy (e.g., using who for people and which for objects).
Methodology and Datasets
The researchers employed a three-pronged evaluation strategy on BERT, RoBERTa, and ALBERT:
- Probing Classifiers: Logistic regression classifiers were trained on hidden layers to distinguish between grammatical and ungrammatical RC sentences.
- Diagnostic Evaluation: Probing classifiers were tested on a "diagnostic set" of 32 hand-crafted sentences designed to challenge models with long-distance dependencies and agreement attractors.
- Masked Prediction (MLM) Task: The models' native ability to fill in blanks was used to test if their top predictions respected grammatical and semantic rules when relativizers or antecedents were masked.
The study utilized a new Naturalistic Probing Dataset of 48,060 sentences from the fiction portion of the COCA corpus (Corpus of Contemporary American English), alongside a smaller, highly controlled diagnostic set and an MLM evaluation set from magazine and academic registers.
Probing and Layer-wise Performance
Initial results suggest that transformer models are highly effective at capturing the features necessary for grammaticality judgments. All three models showed high linear separability for grammaticality early in their layers, with accuracy peaking between 83% and 85%.
As shown in the figure above, BERT, RoBERTa, and ALBERT significantly outperform static embedding baselines like GloVe and fastText, as well as a simple rule-based baseline. This suggests that the contextual representations in MLMs encode substantial information about relative clause structure.
Deceptive Accuracy and Linguistic Nuance
Despite high overall scores, a closer look reveals that models often rely on "short-cut learning." For instance, models might learn a simple heuristic: "if the sentence contains who, it's likely grammatical." Furthermore, performance varies drastically depending on the specific linguistic challenge.
The table above highlights these discrepancies. While models excel at identifying the need for a relativizer (high accuracy on "relativizer omission"), they struggle with specific substitutions. For example, the relativizer which is particularly difficult to predict because that is a frequent and valid substitute in restrictive clauses, leading models to prefer the more common word.
Key Insights
- Grammar vs. Semantics: Models generally handle structural grammar better than semantic plausibility. They can identify the structural requirement for a relativizer but struggle to choose the semantically correct antecedent when multiple options are present.
- Animacy Constraints: RoBERTa was nearly perfect at respecting animacy (e.g., avoiding who for inanimate objects), while ALBERT struggled with human-describing nouns like family or people.
- Model Personalities: BERT is the most precise at predicting exact target words. RoBERTa captures the best general "sense" of constraints but can be overconfident in error cases. ALBERT performs the worst, likely due to its parameter-sharing architecture limiting its capacity for linguistic nuance.
Conclusion
The study concludes that high probing accuracy can be deceptive. Models often rely on superficial cues, such as the presence of a comma to identify non-restrictive clauses, rather than a deep structural understanding. To build a more accurate picture of model knowledge, future evaluations must move beyond simple probing and incorporate diverse diagnostic and masked prediction tasks.
Summary updated