SEAT: Stable and Explainable Attention

Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, Di Wang

Introduction

In Natural Language Processing (NLP), the attention mechanism is widely regarded as the gold standard for model explainability. By visualizing attention weights as "heat maps," researchers can identify which specific words (e.g., "spectacular" in a movie review) drive a model's classification. However, recent research has exposed a "fragility crisis": attention is often notoriously unstable. Minor changes, such as a different random seed during training or swapping a single word for a synonym, can cause the attention heat map to shift entirely, even if the final prediction remains the same. This inconsistency undermines the reliability of attention as a faithful explanation of model logic.

To address this, researchers have developed SEAT (Stable and Explainable ATtention), a novel framework designed to ensure that model explanations remain consistent and reliable under perturbations.

Methodology

SEAT treats the search for a stable explanation as a constrained optimization problem. A stable and explainable attention mechanism must satisfy three core mathematical properties:

  1. Closeness of Prediction: The SEAT-enhanced model should maintain the predictive behavior of the original vanilla model.
  2. Similarity of Explainability: The most influential words (top-$k$ indices) identified by SEAT should largely overlap with those found by the vanilla model.
  3. Stability: Small perturbations in input text or embeddings should not cause significant shifts in either the prediction or the attention distribution.

The authors implement this via a min-max optimization objective. An internal "adversary" (the max part) searches for perturbations that maximize instability, while the model (the min part) is trained to remain resilient against these shifts. To handle the non-differentiable nature of "top-$k$ overlap," a surrogate loss function was developed to allow for efficient training via standard gradient descent.

Experimental Evaluation

SEAT was evaluated across four diverse text classification benchmarks: SST (Stanford Sentiment Treebank), Emotion Recognition, Hate Speech Detection, and Rotten Tomatoes (RottenT). The framework was tested using three common architectures: RNN, BiLSTM, and BERT.

Key Results

SEAT achieves significant improvements in stability without sacrificing predictive performance.

Comparison of attention stability between baseline models and SEAT across four datasets.

As shown in the figure above, vanilla attention distributions (top row) exhibit high divergence across different conditions, characterized by "heavy tails" in the violin plots. In contrast, SEAT (bottom row) effectively "squeezes" these distributions toward zero divergence. For instance, on the Emotion dataset, SEAT reduced the Jensen-Shannon Divergence (JSD)—a measure of attention map volatility—from approximately 0.6 to near-zero levels (e.g., $1.72 \times 10^{-9}$).

Furthermore, SEAT consistently outperformed six baseline methods on interpretability metrics such as Comprehensiveness and Sufficiency, which evaluate whether the identified "important" words truly drive the model's output. Crucially, these gains come with almost zero degradation in F1 scores, proving that the "utility tax" often associated with model constraints is optional.

Conclusion and Future Work

SEAT provides a robust path toward making neural network explanations truly reliable. By optimizing for stability, it ensures that the "why" behind a model's decision is a steady reflection of its logic rather than an artifact of random noise.

Future work will focus on reducing the computational overhead of the min-max optimization loop and exploring the application of SEAT to generative tasks in Large Language Models (LLMs), where explainability remains a complex challenge.

Summary updated

Abstract

Attention mechanism has become a standard fixture in many state-of-the-art natural language processing (NLP) models, not only due to its outstanding performance, but also because it provides plausible innate explanations for neural architectures. However, recent studies show that attention is unstable against randomness and perturbations during training or testing, such as random seeds and slight perturbation of embeddings, which impedes it from being a faithful explanation tool. Thus, a natural question is whether we can find an alternative to vanilla attention, which is more stable and could keep the key characteristics of the explanation. In this paper, we provide a rigorous definition of such an attention method named SEAT (Stable and Explainable ATtention). Specifically, SEAT has the following three properties: (1) Its prediction distribution is close to the prediction of the vanilla attention; (2) Its top-k indices largely overlap with those of the vanilla attention; (3) It is robust w.r.t perturbations, i.e., any slight perturbation on SEAT will not change the attention and prediction distribution too much, which implicitly indicates that it is stable to randomness and perturbations. Furthermore, we propose an optimization method for obtaining SEAT, which could be considered as revising the vanilla attention. Finally, through intensive experiments on various datasets, we compare our SEAT with other baseline methods using RNN, BiLSTM and BERT architectures, with different evaluation metrics on model interpretation, stability and accuracy. Results show that, besides preserving the original explainability and model performance, SEAT is more stable against input perturbations and training randomness, which indicates it is a more faithful explanation.
BibTeX
@inproceedings{hu2023seat,
  author = {Lijie Hu and Yixin Liu and Ninghao Liu and Mengdi Huai and Lichao Sun and Di Wang},
  title = {{SEAT: Stable and Explainable Attention}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26517},
}
Figure