Analyzing the Structure of Attention in a Transformer Language Model

Jesse Vig, Yonatan Belinkov

Introduction

While Transformer-based models like GPT-2 have fundamentally changed the landscape of Natural Language Processing, they are frequently characterized as "black boxes." We observe their state-of-the-art performance, yet the internal mechanics of their self-attention layers remain largely opaque. This research seeks to look under the hood of GPT-2 to determine if its attention mechanism encodes human-like linguistic structure or if it merely relies on statistical artifacts.

By visualizing attention at the head, model, and neuron levels, this study demonstrates that attention is a structured system. Specific heads are shown to specialize in distinct linguistic tasks, such as identifying parts of speech, resolving grammatical dependencies, and bridging long-distance context.

Methods and Visualization

The researchers employed a dual approach combining interactive visualization with aggregate statistical analysis to inspect GPT-2's internal state. They introduced three levels of granularity for analysis:

  1. Attention-head view: Visualizes how tokens in a specific head attend to one another.
  2. Model view: Provides a high-level grid of thumbnails showing the "shape" of attention across all layers and heads simultaneously.
  3. Neuron view: A deep dive into how individual neurons within the query and key vectors interact to produce the final attention score.
The model view provides a high-level grid of attention patterns across all layers and heads.

To move beyond qualitative observations, the study defined several quantitative metrics to evaluate attention patterns:

  • Dependency Alignment: Measures how often attention connects words with formal grammatical relationships.
  • Attention Variability: Distinguishes between "position-based" heads (e.g., looking at the previous word) and "content-based" heads (searching for specific meanings).
  • Attention Distance: Calculates the average span between tokens that attend to each other.

Datasets

The analysis was conducted using text from English Wikipedia. While GPT-2 was originally trained on the WebText dataset, the authors utilized a sample of 100,000 sentences from Wikipedia to evaluate general linguistic patterns, with a subset of 10,000 sentences used for the final quantitative analysis.

Key Insights: Functional Specialization and Depth

The study revealed several critical insights into how Transformers process language across their layers:

The "Middle Layer" Sweet Spot

Grammatical dependency alignment is strongest in the middle layers of the model. This suggests a hierarchical processing pipeline: the model first processes basic features in shallow layers, constructs a syntactic map in the middle layers, and finally shifts to abstract or task-specific patterns in the deepest layers.

Functional Specialization

Attention heads are highly specialized. Some focus on punctuation, others on specific parts of speech, and many exhibit "null attention"—defaulting to the first token of a sentence when no relevant relationship is found. On average, 57% of attention is directed to the first token, acting as a "no-op" state.

Comparison of attention heads demonstrating different linguistic functions and quantitative metrics.

Depth vs. Distance

There is a clear correlation between layer depth and attention distance. Shallow layers tend to look at immediate neighbors, while deeper layers span a greater number of tokens to find connections. There is a moderate correlation ($r = 0.61$) between how "diffuse" (spread out) an attention pattern is and the distance it covers, though some "sharp" heads can jump long distances to find repeated phrases.

Key Results

  • Syntax Alignment: Middle-layer heads (such as Layer 4, Head 11) showed remarkably high alignment with dependency relations, often attending to the previous token with high precision.
  • POS Targeting: Heads targeting "proper nouns" were primarily located in the final three layers, suggesting that identifying named entities requires deep context. Conversely, heads targeting "determiners" (like 'the' or 'a') appeared in the first four layers.

Limitations and Future Work

This study focused on the GPT-2 small model, a decoder-only architecture. Future research is needed to determine if these patterns hold in bidirectional models like BERT or in significantly larger versions of GPT. Additionally, as the results are based on formal Wikipedia text, attention patterns might manifest differently in informal contexts like dialogue. Finally, exploring attention across much longer documents remains a vital area for future investigation to test the true limits of long-distance Transformer capabilities.

Summary updated

Abstract

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.
BibTeX
@inproceedings{vig2019analyzing,
  author = {Jesse Vig and Yonatan Belinkov},
  title = {{Analyzing the Structure of Attention in a Transformer Language Model}},
  booktitle = {BlackboxNLP @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=taKZVj94As},
}
Figure