In the quest to open the "black box" of Natural Language Processing (NLP), researchers have long leaned on attention mechanisms as a built-in window into a model's reasoning. If a model predicts that a movie review is negative and the attention weights are high on the word "awful," we assume the model is behaving logically.
However, recent evidence suggests that this interpretation is often an illusion. Raw attention weights tell us which tokens a model considers important, but they fail to communicate the direction of that importance. This fundamental gap between importance and influence leads to explanations that are unfaithful to the model's actual decision-making process.
The Importance vs. Influence Trap
The central problem with using raw attention as an explanation is its lack of polarity. As André Panisson, Francesco Bodria, and their colleagues point out in Explainability Methods for Natural Language Processing: Applications to Sentiment Analysis, attention scores identify influential tokens but cannot distinguish whether a token contributes to a positive or negative prediction.
Imagine a sentiment classifier processing the sentence: "The movie was not good." A model might place high attention on the word "good." Without polarity information, a human observer might conclude the model is mistakenly focusing on a positive word to make a negative prediction. In reality, the model might be using the word "good" specifically to suppress the positive class score. Raw attention conflates these two opposite behaviors—supporting a class and suppressing it—into a single magnitude.
The Failure of Standard Metrics
To measure if an explanation is "faithful," researchers typically use metrics like sufficiency (does the model reach the same conclusion using only the highlighted tokens?) and comprehensiveness (does the model's confidence drop when the highlighted tokens are removed?).
However, Haoliang Li, Yibing Liu, and their team argue in Rethinking Attention-Model Explainability through Faithfulness Violation Test that these metrics are fundamentally flawed. Because these metrics average performance across many tokens, they mask "polarity flips."
If an explanation method highlights a feature that actually suppresses the correct class, a standard metric might still give it a high score if other highlighted features are supportive. This creates a "faithfulness violation" where the explanation claims a feature is a primary reason for a decision, even though removing that feature would actually make the model more confident in its prediction.
Detecting Faithfulness Violations
To address this, Li et al. introduced the Faithfulness Violation Test (FVT). This diagnostic tool specifically looks for instances where the polarity of a feature's influence is misrepresented.
The FVT reveals a striking reality: many features identified as "highly important" by raw attention are actually working against the model's final decision. When these suppressing features are removed, the model’s confidence in the correct class increases—the opposite of what a faithful explanation should imply.
Moving Toward Gradient-Based Explanations
If raw attention is an unreliable narrator, what should we use instead? The research points toward gradient-based methods. Unlike raw attention weights, which are purely based on the similarity between hidden states, gradient-based methods (like Integrated Gradients or Grad-Attention) account for how a change in the input actually affects the output probability.
By incorporating the gradient, these methods naturally capture polarity. They can distinguish between a feature that pushes the probability up (support) and one that pulls it down (suppression). As demonstrated by the work of Li et al., these gradient-aware interpretations are significantly more reliable and pass faithfulness tests that raw attention fails.
The shift in the field is clear: to truly understand a model, we must look beyond where it is "looking" and start measuring what it is "doing" with that information.