Abstract
Attention mechanisms in Natural Language Processing (NLP) are frequently utilized to provide post-hoc explanations for model decisions. However, these mechanisms are often unstable; minor perturbations in input or changes in random seeds can lead to significantly different attention maps, undermining their reliability. This paper introduces SEAT (Stable and Explainable Attention), a framework designed to enforce consistency, faithfulness, and accuracy in attention-based explanations. By employing min-max optimization and a novel differentiable top-$k$ surrogate loss, SEAT ensures that attention maps remain stable under pressure. Our evaluation across diverse architectures and benchmarks demonstrates that SEAT achieves substantial stability gains while maintaining or improving predictive performance.
Introduction: The Fragility of Attention
In modern NLP, the attention mechanism is a cornerstone of model architecture, serving both as a performance booster and a diagnostic tool. By highlighting specific tokens, attention maps offer a "window" into which parts of the input the model prioritized. Despite their popularity, recent research has exposed a "trust crisis": attention is fragile. Small changes, such as swapping a word for a synonym or varying the training seed, can cause attention weights to shift drastically. This instability suggests that the provided explanations may be artifacts of noise rather than true representations of the model's reasoning. SEAT addresses this by asking: Can we build an attention mechanism that is resilient to noise while remaining faithful to the model's predictions?
Methodology: The SEAT Framework
The authors define SEAT through three fundamental mathematical properties:
- Closeness of Prediction: The output of the SEAT-enhanced model must remain nearly identical to that of the original "vanilla" attention model.
- Similarity of Explainability: The most critical tokens (the top-$k$ indices) should remain consistent with those identified by the original model.
- Stability: Predictions should not fluctuate due to minor nudges or noise in the attention weights.
Technical Implementation
SEAT frames the challenge as a min-max stochastic optimization problem. The goal is to minimize prediction variance while maximizing the model's tolerance to perturbations. A significant technical hurdle is that "top-$k$ ranking" is inherently non-differentiable, preventing standard gradient-based optimization. To overcome this, the authors developed a Top-$k$ Overlap Surrogate Loss, allowing the model to learn stable feature importance via gradient descent. Additionally, Projected Gradient Descent (PGD) is used to identify "worst-case" noise, training the model to be robust against adversarial-like perturbations.
Experimental Evaluation
The researchers evaluated SEAT across three architectures—RNN, BiLSTM, and BERT—using four text classification benchmarks:
- SST (Stanford Sentiment Treebank): Movie review sentiment analysis.
- Emotion: Multi-class emotion detection.
- Hate: Hate speech identification.
- RottenT (Rotten Tomatoes): Sentiment analysis.
- IMDB: Used for qualitative heat map visualizations.
Key Results and Insights
SEAT significantly outperforms vanilla attention and existing robust baselines (e.g., Word-AT, Att-AT) across several dimensions.
Stability and Consistency
The most striking result is the improvement in stability. On the BERT/SST dataset, the Jensen-Shannon Divergence (JSD)—a metric for attention map shifts—was reduced from $0.005$ in the vanilla model to nearly zero ($2.77 \times 10^{-7}$) with SEAT. Similarly, the Total Variation Distance (TVD) for predictions was significantly lower, indicating that SEAT is less likely to change its output due to minor input noise.
Interpretability and Performance
Unlike many regularization techniques that impose an "accuracy tax," SEAT maintains or slightly improves F1 scores. Furthermore, metrics like "Comprehensiveness" and "Sufficiency" confirm that the tokens highlighted by SEAT are indeed the ones the model relies on for its final decision, ensuring the explanations are both stable and faithful.
Conclusion and Future Directions
SEAT represents a major step toward reliable NLP explanations. By prioritizing the stability of the "Top-k" most important words, it provides human-readable explanations that remain consistent across perturbations.
While effective, the authors note several areas for future work. Currently, SEAT is evaluated primarily on binary classification; extending it to complex tasks like multi-choice Question Answering (QA) is a priority. Additionally, the fixed hyperparameter $k$ (set to 7 in these tests) may need dynamic adjustment for varying document lengths. Finally, integrating these stability constraints into the pre-training phase of large-scale Transformers could further enhance the robustness of modern NLP systems.
Summary updated