Analyzing the Structure of Attention in a Transformer Language Model

Jesse Vig, Yonatan Belinkov

Abstract

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.

Introduction

Transformer-based models like GPT-2 and BERT have revolutionized Natural Language Processing (NLP), yet they remain notoriously difficult to interpret. While their performance is undeniable, the internal mechanisms by which they represent the complexities of human syntax and semantics are not fully understood. This study aims to bridge this gap by investigating whether the mathematical attention weights in a Transformer correspond to recognizable linguistic concepts, such as parts of speech (POS) or dependency relations.

Methodology

The researchers employed a two-pronged approach to dissect the GPT-2 small model:

  1. Multi-Level Visualization: Utilizing the BertViz tool, the authors visualized attention at three scales: the attention-head level (token-to-token interactions), the model level (a global view of all layers and heads), and the neuron level (interactions within query and key vectors).
  2. Aggregate Analysis: To move beyond anecdotal evidence, statistical metrics were calculated across a large corpus. These included Dependency Alignment (measuring coincidence with syntactic dependencies), Attention Variability (distinguishing position-based from content-based heads), and Attention Distance (the average token span of attention).
A model-level visualization providing a birds-eye view of attention patterns across multiple layers and heads.

Data

The study utilized the GPT-2 small pretrained model and evaluated its behavior using text from English Wikipedia. A sample of 100,000 sentences was used for qualitative analysis, while a subset of 10,000 sentences provided the basis for quantitative metrics. This choice of dataset ensures that the analysis reflects how the model generalizes to text outside its original training distribution.

Analysis of Attention Patterns

The "First Token" Sink

A significant finding was that, on average, 57% of attention is directed to the first token of a sentence. This suggests the first token acts as a "null" state or a placeholder when an attention head does not find a relevant target for its specific linguistic task.

Layer Specialization and Distance

The model exhibits a clear functional hierarchy across its layers:

  • Early Layers: Focus on low-level constructs and local patterns, such as determiners (e.g., "the", "a").
  • Middle Layers: Act as the "syntactic engine," showing the strongest alignment with dependency relations (Layers 4-9).
  • Deep Layers: Handle high-level abstractions and the longest-distance relationships.
Comparison of attention heads showing short-distance/content-independent behavior (left) versus long-distance/content-dependent behavior (right).

Position vs. Content

Attention heads vary from purely mechanical to highly specialized. For example, some heads are "position-based," simply attending to the previous token regardless of content. Others are "content-based" detectors that target specific linguistic features, such as acronyms (linking "NASA" to its full name), commas in place names, or the ends of noun phrases.

Key Results

  • Syntactic Alignment: Dependency alignment peaks in the middle layers, where the model most closely mirrors human-defined grammar.
  • Increasing Depth, Increasing Distance: The average attention distance grows with layer depth, indicating that deeper layers are responsible for integrating information across broader contexts.
  • POS Specificity: Certain heads are remarkably specialized; one head in Layer 9 directs 43% of its attention to nouns, more than double the model-wide average.

Conclusion and Limitations

This analysis provides a clearer picture of the internal logic of GPT-2, suggesting that attention is a structured, linguistically relevant mechanism rather than a random weight distribution. However, the findings are specific to the GPT-2 small decoder-only architecture and Wikipedia text. Future research is needed to determine if these patterns hold for encoder-only models like BERT or for models processing more informal, non-encyclopedic text.

Summary updated

Abstract

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.
BibTeX
@inproceedings{vig2019analyzing,
  author = {Jesse Vig and Yonatan Belinkov},
  title = {{Analyzing the Structure of Attention in a Transformer Language Model}},
  booktitle = {BlackboxNLP @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=taKZVj94As},
}
Figure