How is BERT surprised? Layerwise detection of linguistic anomalies

Bai Li, Zining Zhu, Guillaume Thomas, Yang Xu, Frank Rudzicz

Introduction

Large Language Models (LLMs) like BERT and RoBERTa demonstrate remarkable proficiency in identifying linguistic anomalies. However, standard evaluation metrics typically rely on final-layer output probabilities, treating the model's internal architecture as a "black box." This approach identifies that an anomaly exists but fails to explain why or when the model detects it. This paper investigates the internal "surprisal" of Transformer models across individual layers to uncover how different linguistic errors—ranging from simple syntax to complex world knowledge—are processed.

Methodology: Probing with Density Estimation

To "open the box," the researchers employed a layerwise density estimation technique. Unlike traditional probes that require training a classifier, this method uses a small sample of "normal" English text (from the British National Corpus) to establish a baseline for each layer.

For every layer in a model (e.g., BERT-base), word embeddings are extracted and fitted to a multivariate Gaussian distribution. This distribution represents the "expected" state of a word at that specific stage of processing. When a test sentence is processed, the model calculates the negative log-likelihood (surprisal) of the new embeddings against the established Gaussian model. A "surprisal gap"—the difference in surprisal between a correct sentence and its anomalous counterpart—serves as the primary metric for sensitivity.

Layerwise surprisal heatmap for a morphosyntactic error ("eating"). Darker colors indicate higher surprisal relative to the baseline distribution.

The Linguistic Pipeline

The study reveals a clear internal hierarchy in how Transformer models process language, mirroring certain event-related potentials (ERPs) observed in human brain activity:

  1. Early Layers (0-2): Surprisal is primarily driven by word frequency. Rare words appear anomalous simply because they are statistically infrequent in the training data.
  2. Middle Layers (3-6): These layers specialize in morphosyntax. Errors in form, such as subject-verb agreement or incorrect verb tense (e.g., "the cat won't eating"), trigger significant surprisal early in the pipeline.
  3. Late Layers (9-11): Semantic anomalies, such as selectional restriction violations (e.g., "the plane laughed"), peak much later. This suggests that the model requires deeper contextual processing to identify meaning-based mismatches.
  4. The Commonsense Gap: Surprisingly, "commonsense" errors (e.g., "the customer served the waitress") trigger almost no internal surprisal at any layer. While the models can eventually distinguish these errors at the final output, their internal representations do not treat them as "out-of-distribution" events, suggesting world knowledge is handled via a different mechanism than grammar or basic semantics.

Quantitative Performance and Efficiency

The Gaussian anomaly detection method proved remarkably effective. RoBERTa-base achieved an accuracy of 0.830 on the BLiMP benchmark, outperforming several baselines that utilize the full output vocabulary.

Performance metrics: (a) Accuracy vs. number of training sentences, showing rapid convergence; (b) Accuracy vs. layer, highlighting model-specific sensitivity peaks.

As shown in the figures above, the method is highly data-efficient, reaching a performance plateau with as few as 1,000 training sentences. Furthermore, the "peak" layer for detection varies by architecture: layer 9 for BERT, layer 11 for RoBERTa, and layer 6 for XLNet.

Conclusion

By analyzing layerwise surprisal, this research demonstrates that Transformer models possess an internal "linguistic pipeline" that separates syntax from semantics. While these models are highly sensitive to structural and meaning-based anomalies, the lack of internal surprisal for commonsense violations highlights a significant area for future architectural exploration. Understanding where and how world knowledge is integrated remains a key challenge for the next generation of LLMs.

Summary updated

Abstract

Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is morphosyntactic than when it is semantic, while commonsense anomalies do not exhibit surprisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of linguistic anomalies.
BibTeX
@inproceedings{li2021how,
  author = {Bai Li and Zining Zhu and Guillaume Thomas and Yang Xu and Frank Rudzicz},
  title = {{How is BERT surprised? Layerwise detection of linguistic anomalies}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.325},
}
Figure