SEAT: Stable and Explainable Attention

Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, Di Wang

Abstract

Attention mechanisms in Natural Language Processing (NLP) are frequently utilized to provide post-hoc explanations for model decisions. However, these mechanisms are often unstable; minor perturbations in input or changes in random seeds can lead to significantly different attention maps, undermining their reliability. This paper introduces SEAT (Stable and Explainable Attention), a framework designed to enforce consistency, faithfulness, and accuracy in attention-based explanations. By employing min-max optimization and a novel differentiable top-$k$ surrogate loss, SEAT ensures that attention maps remain stable under pressure. Our evaluation across diverse architectures and benchmarks demonstrates that SEAT achieves substantial stability gains while maintaining or improving predictive performance.

Introduction: The Fragility of Attention

In modern NLP, the attention mechanism is a cornerstone of model architecture, serving both as a performance booster and a diagnostic tool. By highlighting specific tokens, attention maps offer a "window" into which parts of the input the model prioritized. Despite their popularity, recent research has exposed a "trust crisis": attention is fragile. Small changes, such as swapping a word for a synonym or varying the training seed, can cause attention weights to shift drastically. This instability suggests that the provided explanations may be artifacts of noise rather than true representations of the model's reasoning. SEAT addresses this by asking: Can we build an attention mechanism that is resilient to noise while remaining faithful to the model's predictions?

Methodology: The SEAT Framework

The authors define SEAT through three fundamental mathematical properties:

  1. Closeness of Prediction: The output of the SEAT-enhanced model must remain nearly identical to that of the original "vanilla" attention model.
  2. Similarity of Explainability: The most critical tokens (the top-$k$ indices) should remain consistent with those identified by the original model.
  3. Stability: Predictions should not fluctuate due to minor nudges or noise in the attention weights.

Technical Implementation

SEAT frames the challenge as a min-max stochastic optimization problem. The goal is to minimize prediction variance while maximizing the model's tolerance to perturbations. A significant technical hurdle is that "top-$k$ ranking" is inherently non-differentiable, preventing standard gradient-based optimization. To overcome this, the authors developed a Top-$k$ Overlap Surrogate Loss, allowing the model to learn stable feature importance via gradient descent. Additionally, Projected Gradient Descent (PGD) is used to identify "worst-case" noise, training the model to be robust against adversarial-like perturbations.

Experimental Evaluation

The researchers evaluated SEAT across three architectures—RNN, BiLSTM, and BERT—using four text classification benchmarks:

  • SST (Stanford Sentiment Treebank): Movie review sentiment analysis.
  • Emotion: Multi-class emotion detection.
  • Hate: Hate speech identification.
  • RottenT (Rotten Tomatoes): Sentiment analysis.
  • IMDB: Used for qualitative heat map visualizations.

Key Results and Insights

SEAT significantly outperforms vanilla attention and existing robust baselines (e.g., Word-AT, Att-AT) across several dimensions.

Stability and Consistency

The most striking result is the improvement in stability. On the BERT/SST dataset, the Jensen-Shannon Divergence (JSD)—a metric for attention map shifts—was reduced from $0.005$ in the vanilla model to nearly zero ($2.77 \times 10^{-7}$) with SEAT. Similarly, the Total Variation Distance (TVD) for predictions was significantly lower, indicating that SEAT is less likely to change its output due to minor input noise.

Figure 3: Comparison of attention stability between Baseline and SEAT (Ours) on the SST dataset. The SEAT distributions are significantly more concentrated near zero, indicating higher stability.

Interpretability and Performance

Unlike many regularization techniques that impose an "accuracy tax," SEAT maintains or slightly improves F1 scores. Furthermore, metrics like "Comprehensiveness" and "Sufficiency" confirm that the tokens highlighted by SEAT are indeed the ones the model relies on for its final decision, ensuring the explanations are both stable and faithful.

Figure 2: Stability comparison on the Hate speech dataset. SEAT (Ours-Hate) shows tighter, more consistent distributions across confidence bins compared to the baseline.

Conclusion and Future Directions

SEAT represents a major step toward reliable NLP explanations. By prioritizing the stability of the "Top-k" most important words, it provides human-readable explanations that remain consistent across perturbations.

While effective, the authors note several areas for future work. Currently, SEAT is evaluated primarily on binary classification; extending it to complex tasks like multi-choice Question Answering (QA) is a priority. Additionally, the fixed hyperparameter $k$ (set to 7 in these tests) may need dynamic adjustment for varying document lengths. Finally, integrating these stability constraints into the pre-training phase of large-scale Transformers could further enhance the robustness of modern NLP systems.

Summary updated

BibTeX
@inproceedings{hu2023seat,
  author = {Lijie Hu and Yixin Liu and Ninghao Liu and Mengdi Huai and Lichao Sun and Di Wang},
  title = {{SEAT: Stable and Explainable Attention}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26517},
}
Figure