A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English

Marius Mosbach, Stefania Degaetano-Ortlieb, Marie-Pauline Krielke, Badr M. Abdullah, Dietrich Klakow

Introduction: The "BERTology" Problem

While transformer-based models like BERT, RoBERTa, and ALBERT demonstrate impressive performance on various benchmarks, the NLP community faces a "BERTology" problem: we lack a deep understanding of the specific linguistic rules these models have actually mastered. A common evaluation method is "probing," where a simple classifier is trained on a model's frozen representations to detect linguistic features. However, high probing scores can be deceptive, as models might rely on simple heuristics or "shortcuts" rather than genuine structural knowledge.

This study investigates this phenomenon by focusing on Relative Clauses (RCs) in American English. RCs serve as an ideal stress test because they require models to link a relativizer (who, which, that) to an antecedent while respecting both complex grammatical rules and semantic constraints like animacy.

Methodology

The researchers employed a three-pronged strategy to evaluate linguistic depth:

  1. Probing Classifiers: Logistic regression classifiers were trained on sentence embeddings to distinguish between grammatical and ungrammatical RCs using a naturalistic dataset of ~48,000 sentences from the Corpus of Contemporary American English (COCA).
  2. Diagnostic Cases: A targeted "stress test" set of 32 sentences evaluated challenges such as long-distance dependencies and "agreement attractors" (distractor nouns that might trigger incorrect relativizer choices).
  3. Masked Language Modeling (MLM) Task: The models' native "fill-in-the-blank" capabilities were tested by masking relativizers or antecedents to see if predictions remained grammatically and semantically plausible.

Results: The Probing Illusion

All three models achieved high probing accuracy, ranging from 83.0% to 85.5%. However, non-contextualized baselines like GloVe also performed surprisingly well, suggesting that the models may be exploiting simple surface-level patterns (e.g., the mere presence of the word who) rather than deep syntactic structures.

Probing accuracy across different relative clause modifications for various models.

As shown in the table above, while the transformer models outperform baselines on the total score, specific modifications like "which → who" reveal significant drops in performance for some models, indicating a fragile grasp of certain semantic constraints.

Layer-wise Analysis and Model "Personalities"

Analysis of the models' internal layers shows that linguistic knowledge for RCs emerges very early, typically between layers 1 and 5, before plateauing in the higher layers.

Layer-wise probing accuracy for relative clauses across different models.

The study also identified distinct "personalities" among the models:

  • RoBERTa acts as a "Semantic Specialist," showing high confidence in detecting agreement errors but extreme sensitivity to sentence length and distance.
  • BERT and ALBERT are more "Structuralist," handling long-distance dependencies better than RoBERTa but proving more susceptible to confusion from agreement attractors.

Conclusion

The findings suggest that models are generally better at following strict grammatical rules (like subject-verb agreement) than nuanced semantic ones (like animacy). Probing alone tends to overestimate true linguistic "understanding." Future evaluations should combine probing with diagnostic tests and native task evaluations to provide a more accurate picture of a model's capabilities. Expanding this research to larger model scales and non-English languages remains a critical next step.

Summary updated

Abstract

Transformer-based language models achieve high performance on various tasks, but we still lack understanding of the kind of linguistic knowledge they learn and rely on. We evaluate three models (BERT, RoBERTa, and ALBERT), testing their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. We focus on relative clauses (in American English) as a complex phenomenon needing contextual information and antecedent identification to be resolved. Based on a naturalistic dataset, probing shows that all three models indeed capture linguistic knowledge about grammaticality, achieving high performance.Evaluation on diagnostic cases and masked prediction tasks considering fine-grained linguistic knowledge, however, shows pronounced model-specific weaknesses especially on semantic knowledge, strongly impacting models’ performance. Our results highlight the importance of (a)model comparison in evaluation task and (b) building up claims of model performance and the linguistic knowledge they capture beyond purely probing-based evaluations.
BibTeX
@inproceedings{mosbach2020closer,
  author = {Marius Mosbach and Stefania Degaetano-Ortlieb and Marie-Pauline Krielke and Badr M. Abdullah and Dietrich Klakow},
  title = {{A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.67},
}
Figure