Understanding Attention for Text Classification

Xiaobing Sun, Wei LuACL 2020 · DOI

Abstract

This paper investigates the interpretability of attention mechanisms in text classification by analyzing the internal gradient update process during training. Moving beyond the "attention is explanation" debate, the authors introduce global metrics—attention scores and polarity scores—to characterize word-label associations. Through gradient flow analysis, they demonstrate that while local attention weights can be noisy, these global scores drive model behavior. Crucially, the scaling hyperparameter $\lambda$ acts as an "interpretability knob," where high values maintain accuracy but degrade the human-readability of attention weights.

Introduction

A central debate in Natural Language Processing (NLP) concerns whether attention weights provide faithful explanations for model decisions. While some evidence suggests a correlation between attention and feature importance, other studies show that attention weights can be significantly altered without impacting model output. This paper addresses this discrepancy by shifting focus from "local" attention weights (specific to a single instance) to the "global" evolution of the attention mechanism during training. By examining why specific tokens like "excellent" or "terrible" attract high attention in certain configurations, the authors aim to clarify the relationship between attention, model performance, and interpretability.

Methodology

The study analyzes a standard neural text classification architecture employing both scaled dot-product and additive attention. The model processes input embeddings through an attention layer, followed by linear and sigmoid transformations to produce a final classification output.

Figure 1: The standard neural text classification architecture analyzed in the study, featuring an attention layer followed by linear and sigmoid transformations.

The authors utilize gradient flow analysis to track the development of two key global quantities:

  1. Attention Score ($a_e$): An instance-independent measure of a token's absolute significance across the entire corpus.
  2. Polarity Score ($s_e$): A measure of the strength of association between a word and a specific class (e.g., positive vs. negative).

The theoretical framework was evaluated on four benchmarks: SST (binary sentiment), IMDB (large-scale sentiment), and two subsets of the 20 Newsgroups dataset (binary and 3-class classification).

Key Insights: The Role of Scaling and Polarity

A primary finding is that the scaling factor $\lambda$ in dot-product attention serves as a critical "interpretability knob." While models achieve peak accuracy at moderate $\lambda$ values, they remain remarkably robust even at extremely high values (e.g., $\lambda=10,000$). However, at these high values, attention distributions become nearly uniform (approaching mean pooling), rendering them uninterpretable to humans despite high classification accuracy. This suggests that accuracy and interpretability are not always coupled; the model relies on underlying polarity scores even when the attention mechanism appears non-selective.

Results and Visualization

The theoretical gradient analysis is supported by empirical visualizations across various datasets. As shown in Figure 3, tokens with high positive or negative sentiment (or class-specific relevance) cluster at the extremes of the polarity spectrum.

Figure 3: Scatter plots showing the distribution of Polarity Scores and Attention Scores across different token indices for various datasets, confirming the clustering of high-polarity tokens.

These high-polarity words naturally "win" the gradient competition during training, resulting in higher global attention scores compared to neutral "filler" words. This consistency holds across various encoders, including standard embeddings and LSTMs, confirming that polarity is a primary driver of the attention mechanism's focus under ideal hyperparameter settings.

Conclusion and Future Directions

This work provides a theoretical foundation for understanding how attention mechanisms learn word importance. By distinguishing between local weights and global scores, the authors clarify why attention can sometimes appear misleading. Future research is needed to extend this gradient flow analysis to more complex, multi-layered architectures like Transformers and to investigate the impact of pre-trained or contextualized embeddings on the evolution of these scores.

Summary updated

BibTeX
@inproceedings{sun2020understanding,
  author = {Xiaobing Sun and Wei Lu},
  title = {{Understanding Attention for Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.312},
}
Figure