Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT

Soo RyuEMNLP 2023 · DOI

Abstract

This research investigates the mechanisms by which Transformer-based language models, specifically GPT-2, process semantic "world knowledge." By analyzing the plausibility of noun-verb relationships, we identify specific attention heads that function as semantic filters. Our findings demonstrate that a small fraction of these heads are responsible for the model's human-like processing of meaning. Through surprisal analysis and causal pruning, we reveal that GPT-2's incremental processing closely mirrors human reading patterns and that a single head in the initial layer acts as a critical bottleneck for semantic sensitivity.

Introduction

While Transformer models have demonstrated mastery over linguistic syntax, their capacity for managing semantic knowledge—the "common sense" required to determine if a sentence is plausible—remains a subject of intense study. For instance, a model must distinguish between the plausible "The butler shattered the plate" and the implausible "The butler shattered the letter." This paper moves beyond evaluating final model outputs to investigate the internal localization of this knowledge: is it distributed across the architecture, or do specific attention heads specialize in semantic plausibility?

Methods

The study employed three primary techniques to dissect GPT-2:

  1. Surprisal Analysis: Measuring "surprisal" (the unexpectedness of a word) to compare GPT-2, BERT, RoBERTa, and ALBERT against human reading time data from the Cunnings and Sturt (2018) psycholinguistic dataset.
  2. Head-Level Accuracy Probing: Defining a metric to determine if individual attention heads "prefer" plausible nouns over implausible ones based on attention focus at the verb.
  3. Causal Pruning: Systematically zeroing out specific heads to measure their impact on model performance and plausibility sensitivity, using Andersen’s "The Money Box" to monitor general perplexity.

Results and Discussion

Human Similarity and Architecture

A key finding is that architecture significantly influences how models align with human cognition. GPT-2, a decoder-only model, exhibited a significantly higher correlation with human reading data ($p < .001$) compared to bidirectional models like BERT or RoBERTa. This suggests that GPT-2’s incremental, left-to-right processing more accurately reflects human reading behavior.

Comparison of human reading times and model surprisal across plausibility conditions.

The "Plausibility 18" and Causal Impact

Out of 144 heads in GPT-2 (small), only 18 were identified as highly accurate at detecting plausibility. These heads are diffusely distributed across all 12 layers. To prove their functional necessity, we performed causal pruning. Removing these 18 heads caused the model's "plausibility sensitivity"—the surprisal gap between sensible and nonsensical sentences—to nearly vanish.

Causal pruning results showing the impact of removing plausibility-specific heads versus random heads in GPT-2.

The Generalist Bottleneck: Head (0, 10)

The research highlights the disproportionate importance of Head (0, 10) in the first layer. Although not the most accurate head in isolation, its removal caused a massive drop in plausibility processing and a spike in general perplexity (increasing average surprisal by 2.79 bits). This suggests that certain "generalist" heads provide foundational information that "specialist" heads later refine. Notably, a head's individual accuracy does not always correlate with its causal importance; some accurate heads are redundant, while less accurate ones may be structurally vital.

Conclusion

The study concludes that GPT-2 processes semantic plausibility in a manner that mirrors human patterns, driven by a small subset of specialized attention heads. The identification of critical bottlenecks like Head (0, 10) suggests that understanding AI requires mapping the specific internal pathways through which knowledge exerts influence. Future research should extend these findings to larger models and explore the interactions between attention heads and feed-forward networks.

Summary updated

BibTeX
@inproceedings{ryu2023plausibility,
  author = {Soo Ryu},
  title = {{Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.27},
}
Figure