Introduction
While Transformer-based models like GPT-2 have fundamentally changed the landscape of Natural Language Processing, they are frequently characterized as "black boxes." We observe their state-of-the-art performance, yet the internal mechanics of their self-attention layers remain largely opaque. This research seeks to look under the hood of GPT-2 to determine if its attention mechanism encodes human-like linguistic structure or if it merely relies on statistical artifacts.
By visualizing attention at the head, model, and neuron levels, this study demonstrates that attention is a structured system. Specific heads are shown to specialize in distinct linguistic tasks, such as identifying parts of speech, resolving grammatical dependencies, and bridging long-distance context.
Methods and Visualization
The researchers employed a dual approach combining interactive visualization with aggregate statistical analysis to inspect GPT-2's internal state. They introduced three levels of granularity for analysis:
- Attention-head view: Visualizes how tokens in a specific head attend to one another.
- Model view: Provides a high-level grid of thumbnails showing the "shape" of attention across all layers and heads simultaneously.
- Neuron view: A deep dive into how individual neurons within the query and key vectors interact to produce the final attention score.
To move beyond qualitative observations, the study defined several quantitative metrics to evaluate attention patterns:
- Dependency Alignment: Measures how often attention connects words with formal grammatical relationships.
- Attention Variability: Distinguishes between "position-based" heads (e.g., looking at the previous word) and "content-based" heads (searching for specific meanings).
- Attention Distance: Calculates the average span between tokens that attend to each other.
Datasets
The analysis was conducted using text from English Wikipedia. While GPT-2 was originally trained on the WebText dataset, the authors utilized a sample of 100,000 sentences from Wikipedia to evaluate general linguistic patterns, with a subset of 10,000 sentences used for the final quantitative analysis.
Key Insights: Functional Specialization and Depth
The study revealed several critical insights into how Transformers process language across their layers:
The "Middle Layer" Sweet Spot
Grammatical dependency alignment is strongest in the middle layers of the model. This suggests a hierarchical processing pipeline: the model first processes basic features in shallow layers, constructs a syntactic map in the middle layers, and finally shifts to abstract or task-specific patterns in the deepest layers.
Functional Specialization
Attention heads are highly specialized. Some focus on punctuation, others on specific parts of speech, and many exhibit "null attention"—defaulting to the first token of a sentence when no relevant relationship is found. On average, 57% of attention is directed to the first token, acting as a "no-op" state.
Depth vs. Distance
There is a clear correlation between layer depth and attention distance. Shallow layers tend to look at immediate neighbors, while deeper layers span a greater number of tokens to find connections. There is a moderate correlation ($r = 0.61$) between how "diffuse" (spread out) an attention pattern is and the distance it covers, though some "sharp" heads can jump long distances to find repeated phrases.
Key Results
- Syntax Alignment: Middle-layer heads (such as Layer 4, Head 11) showed remarkably high alignment with dependency relations, often attending to the previous token with high precision.
- POS Targeting: Heads targeting "proper nouns" were primarily located in the final three layers, suggesting that identifying named entities requires deep context. Conversely, heads targeting "determiners" (like 'the' or 'a') appeared in the first four layers.
Limitations and Future Work
This study focused on the GPT-2 small model, a decoder-only architecture. Future research is needed to determine if these patterns hold in bidirectional models like BERT or in significantly larger versions of GPT. Additionally, as the results are based on formal Wikipedia text, attention patterns might manifest differently in informal contexts like dialogue. Finally, exploring attention across much longer documents remains a vital area for future investigation to test the true limits of long-distance Transformer capabilities.
Summary updated