Introduction
Large Language Models (LLMs) like BERT and RoBERTa demonstrate remarkable proficiency in identifying linguistic anomalies. However, standard evaluation metrics typically rely on final-layer output probabilities, treating the model's internal architecture as a "black box." This approach identifies that an anomaly exists but fails to explain why or when the model detects it. This paper investigates the internal "surprisal" of Transformer models across individual layers to uncover how different linguistic errors—ranging from simple syntax to complex world knowledge—are processed.
Methodology: Probing with Density Estimation
To "open the box," the researchers employed a layerwise density estimation technique. Unlike traditional probes that require training a classifier, this method uses a small sample of "normal" English text (from the British National Corpus) to establish a baseline for each layer.
For every layer in a model (e.g., BERT-base), word embeddings are extracted and fitted to a multivariate Gaussian distribution. This distribution represents the "expected" state of a word at that specific stage of processing. When a test sentence is processed, the model calculates the negative log-likelihood (surprisal) of the new embeddings against the established Gaussian model. A "surprisal gap"—the difference in surprisal between a correct sentence and its anomalous counterpart—serves as the primary metric for sensitivity.
The Linguistic Pipeline
The study reveals a clear internal hierarchy in how Transformer models process language, mirroring certain event-related potentials (ERPs) observed in human brain activity:
- Early Layers (0-2): Surprisal is primarily driven by word frequency. Rare words appear anomalous simply because they are statistically infrequent in the training data.
- Middle Layers (3-6): These layers specialize in morphosyntax. Errors in form, such as subject-verb agreement or incorrect verb tense (e.g., "the cat won't eating"), trigger significant surprisal early in the pipeline.
- Late Layers (9-11): Semantic anomalies, such as selectional restriction violations (e.g., "the plane laughed"), peak much later. This suggests that the model requires deeper contextual processing to identify meaning-based mismatches.
- The Commonsense Gap: Surprisingly, "commonsense" errors (e.g., "the customer served the waitress") trigger almost no internal surprisal at any layer. While the models can eventually distinguish these errors at the final output, their internal representations do not treat them as "out-of-distribution" events, suggesting world knowledge is handled via a different mechanism than grammar or basic semantics.
Quantitative Performance and Efficiency
The Gaussian anomaly detection method proved remarkably effective. RoBERTa-base achieved an accuracy of 0.830 on the BLiMP benchmark, outperforming several baselines that utilize the full output vocabulary.
As shown in the figures above, the method is highly data-efficient, reaching a performance plateau with as few as 1,000 training sentences. Furthermore, the "peak" layer for detection varies by architecture: layer 9 for BERT, layer 11 for RoBERTa, and layer 6 for XLNet.
Conclusion
By analyzing layerwise surprisal, this research demonstrates that Transformer models possess an internal "linguistic pipeline" that separates syntax from semantics. While these models are highly sensitive to structural and meaning-based anomalies, the lack of internal surprisal for commonsense violations highlights a significant area for future architectural exploration. Understanding where and how world knowledge is integrated remains a key challenge for the next generation of LLMs.
Summary updated