Abstract
We take a deep look into the behavior of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explaining a model's behavior, we show that attention distributions can nevertheless provide insights into the local behavior of attention heads. This way, we propose a distinction between local patterns revealed by attention and global patterns that refer back to the input, and analyze BERT from both angles. We use gradient attribution to analyze how the output of an attention attention head depends on the input tokens, effectively extending the local attention-based analysis to account for the mixing of information throughout the transformer layers. We find that there is a significant discrepancy between attention and attribution distributions, caused by the mixing of context inside the model. We quantify this discrepancy and observe that interestingly, there are some patterns that persist across all layers despite the mixing.
Introduction
In Natural Language Processing, the Transformer architecture—and specifically BERT—has become the standard. To interpret these models, researchers often rely on attention maps, which visualize the weights assigned to tokens during processing. However, a debate has emerged: "Attention is not Explanation." The core issue is information mixing: as data passes through layers, hidden representations become a "soup" of context. By Layer 10, a token no longer represents a single word but an aggregate of the entire sentence. This study bridges the gap between local behavior (layer-specific attention) and global aggregation (the influence of original input words).
Methodology: Hidden Token Attribution (HTA)
To quantify information flow, the authors use Hidden Token Attribution (HTA). Unlike raw attention, HTA uses gradients to measure how changes in an input vector affect hidden representations. The study distinguishes between:
- Previous Layer Contribution: Measures local behavior by looking at how tokens from layer $l-1$ influence layer $l$.
- Input Contribution: Measures global behavior by tracing influence back to the original, non-contextualized input words (Layer 0).
The analysis was performed on the non-finetuned BERT-base model using the MNLI and XNLI datasets, the latter providing cross-lingual validation in English, German, and Arabic.
The Mismatch: Attention vs. Global Attribution
A critical finding is the divergence between what a head attends to locally and what it depends on globally. While attention maps correlate strongly with local attribution (Pearson’s $r > 0.85$), this relationship collapses when looking at global input contributions.
As shown in Figure 5, the median correlation drops below 0.25 in the final layers. This confirms that attention maps in deep layers are poor proxies for understanding which original input words are driving the model's decisions.
The "Great Smoothing" and Temporal Shifts
Locally, BERT exhibits a "human-like" reading pattern. In early layers, heads attend to "past" tokens (left), shifting toward "future" tokens (right) in deeper layers. This suggests a progression from establishing context to predicting subsequent information.
However, global attribution reveals a "Great Smoothing." The sharp temporal biases seen in attention maps largely disappear when traced back to the input. Because hidden states have already mixed context from both directions, the specific directional bias of a single head is neutralized.
Key Insights
- The [SEP] Token as a "Context Bucket": Many heads attend heavily to the [SEP] token. HTA reveals that [SEP] acts as a storage unit for global sentence context, allowing heads to aggregate sequence-wide information.
- Persistence of Segment Blocks: While most patterns smooth out, "block" patterns (attending to all tokens within one sentence of a pair) persist in global attribution, indicating that BERT maintains strong internal boundaries between segments.
- Cross-Lingual Consistency: The past-to-future attention shift is consistent across English, German, and Arabic, suggesting it is a fundamental property of the Transformer architecture.
Conclusion and Limitations
While gradient attribution provides a powerful lens, it remains a first-order approximation and may not capture all non-linear interactions. This work highlights that while attention maps offer insights into local layer logic, they cannot be used as direct explanations for global model behavior due to the profound impact of information mixing. Future research may explore if larger models like GPT-4 exhibit even more aggressive smoothing or if specific heads can be pruned based on their global contribution redundancy.
Summary updated