Why is Attention Not So Attentive?

Bing Bai, Jian Liang, Guanhua Zhang, Hao Li, Kun Bai, Fei Wang

Abstract

Attention-based methods have played important roles in model interpretations, where the calculated attention weights are expected to highlight the critical parts of inputs (e.g., keywords in sentences). However, recent research found that attention-as-importance interpretations often do not work as we expected. For example, learned attention weights sometimes highlight less meaningful tokens like "[SEP]", ",", and ".", and are frequently uncorrelated with other feature importance indicators like gradient-based measures. In this paper, we demonstrate that one root cause of this phenomenon is the combinatorial shortcuts, which means that, in addition to the highlighted parts, the attention weights themselves may carry extra information that could be utilized by downstream models after attention layers. As a result, the attention weights are no longer pure importance indicators. We theoretically analyze combinatorial shortcuts, design one intuitive experiment to show their existence, and propose two methods to mitigate this issue. We conduct empirical studies on attention-based interpretation models. The results show that the proposed methods can effectively improve the interpretability of attention mechanisms.

Introduction: The Interpretability Crisis in Attention

In deep learning, "attention" is often equated with "importance." If a model attends to a specific word in a sentence, we assume that word was the reason for its prediction. However, researchers have noticed a troubling trend: attention weights often highlight meaningless tokens (like periods or commas) and do not always align with other importance measures.

Comparison of attention mechanisms for model interpretability (A) and explaining black-box models (B).

The authors argue that the root cause is a combinatorial shortcut. Because the attention mask is learned alongside the rest of the model, the downstream layers can learn to "read" the pattern of the mask itself. For example, a model might learn to highlight the first word for every positive review and the last word for every negative review. The model achieves high accuracy by simply checking where the highlight is, completely ignoring the content of the words. In this scenario, the attention mechanism has ceased to be an explanation and has instead become just another layer of complex encoding.

Methods: Mitigating Combinatorial Shortcuts

The authors frame the problem through the lens of causal effect estimation, treating the attention mask as a "treatment" and the model output as the "outcome." To fix the bias caused by combinatorial shortcuts, they propose two strategies:

  1. Random Attention Pretraining: The model's downstream components (the parts that process the attended features) are first trained using completely random attention masks. This forces the model to learn an unbiased relationship between features and labels. Once this "decoder" is fixed, a trainable attention layer is introduced to find the most informative features.
  2. Mask-neutral Learning with Instance Weighting: This method uses a statistical technique called instance weighting to decouple the masks from the labels during training. By re-weighting the loss function, the authors encourage the model to reach a state where the distribution of the masks is independent of the labels, preventing the downstream model from using the mask pattern as a predictive shortcut.

Key Insights: The "Cooperation" Trap

The paper reveals that the attention mechanism and the downstream classifier can "cooperate" to minimize loss in ways that destroy interpretability. If the mask carries a signal, the classifier will use it, even if that signal has nothing to do with the actual features.

The Default Token Demo: Illustrating how models use meaningless tokens as combinatorial shortcuts.

In a striking experiment called the Default Token Demo, the authors added meaningless "default tokens" (A, B, C, D) to sentences. Standard models put over 90% of their attention on these meaningless tokens to distinguish between classes, proving that the models prefer shortcuts over actual semantic meaning.

Empirical Evaluation

The authors evaluated their methods on four standard benchmarks: IMDB and Yelp P. for text sentiment classification, and MNIST and Fashion-MNIST for image classification. The primary metric was post-hoc accuracy—how well the model can predict the correct label using only the features selected by the attention mechanism.

  • Significant Improvements: On the IMDB dataset, the baseline L2X model achieved 89.2% post-hoc accuracy. Adding the proposed Pretraining method boosted this to 97.1%.
  • Restoring Meaning: Qualitative results showed that while standard L2X often picked functional words like "and" or "<PAD>" tokens, the proposed methods successfully shifted the attention to sentiment-carrying words like "wonderful," "great," or "compelling."
  • Domain Differences: The methods were highly effective for text but showed more mixed results for images (MNIST), suggesting that the "smoothness" of the feature space affects how easily the model learns these unbiased relationships.

Limitations and Future Work

While effective, random attention pretraining can be computationally expensive due to the massive space of possible feature combinations. Additionally, the instance weighting method relies on specific assumptions about data distribution that can be difficult to tune. Future work could focus on more efficient ways to estimate the "unbiased" expectation of feature importance, perhaps through more sophisticated causal discovery algorithms.

Summary updated

Abstract

Attention-based methods have played important roles in model interpretations, where the calculated attention weights are expected to highlight the critical parts of inputs~(e.g., keywords in sentences). However, recent research found that attention-as-importance interpretations often do not work as we expected. For example, learned attention weights sometimes highlight less meaningful tokens like "[SEP]", ",", and ".", and are frequently uncorrelated with other feature importance indicators like gradient-based measures. A recent debate over whether attention is an explanation or not has drawn considerable interest. In this paper, we demonstrate that one root cause of this phenomenon is the combinatorial shortcuts, which means that, in addition to the highlighted parts, the attention weights themselves may carry extra information that could be utilized by downstream models after attention layers. As a result, the attention weights are no longer pure importance indicators. We theoretically analyze combinatorial shortcuts, design one intuitive experiment to show their existence, and propose two methods to mitigate this issue. We conduct empirical studies on attention-based interpretation models. The results show that the proposed methods can effectively improve the interpretability of attention mechanisms.
BibTeX
@inproceedings{bai2020why,
  author = {Bing Bai and Jian Liang and Guanhua Zhang and Hao Li and Kun Bai and Fei Wang},
  title = {{Why is Attention Not So Attentive?}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2020},
}
Figure