TL;DR
This paper investigates whether comparing different AI explanation methods—specifically attention weights versus feature attribution—is a valid way to evaluate "faithfulness." The authors find that these methods rarely agree on which words are important, especially in Transformer models, leading them to argue that the community should abandon rank correlation as a standard evaluation metric for Explainable AI (XAI).
What Problem Does This Paper Address?
In Natural Language Processing (NLP), we often want to know why a model made a specific prediction. Two popular ways to do this are looking at attention weights (how much the model "focused" on a word) and feature attribution methods (mathematical techniques like LIME or SHAP that calculate a word's contribution).
A heated debate exists: Is attention actually an explanation? Previous influential work argued "no" because attention rankings don't correlate well with gradient-based rankings. This created a paradigm called "agreement as evaluation"—the idea that if two explanation methods don't agree, one must be unfaithful.
The authors of this paper question this paradigm. If two "trusted" attribution methods don't even agree with each other, how can we use one to debunk another?
Methods
The researchers conducted a large-scale empirical study to test the "agreement" hypothesis across different architectures and tasks.
- Models: They compared a traditional Bi-LSTM (with additive attention) against a modern Transformer (DistilBERT).
- Explanation Methods: They tested two attention-based methods (Raw Attention for LSTMs and Attention Rollout for Transformers) against five state-of-the-art feature attribution methods:
- LIME: Perturbs inputs to see how the output changes.
- Integrated Gradients (IG): A gradient-based approach.
- DeepLIFT: Attributes changes in output to input features.
- Grad-SHAP & Deep-SHAP: Two variations of Shapley values, which are rooted in game theory.
- The Metric: They used Kendall-τ (tau) rank correlation, which measures how similarly two methods rank the importance of tokens in a sentence.
Datasets
The study used five diverse datasets covering two types of NLP tasks:
- Single-sequence (Sentiment Analysis): SST-2 and IMDb movie reviews.
- Pair-sequence (Inference/Similarity): MNLI, SNLI (textual entailment), and Quora Question Pairs (identifying duplicate questions).
Key Insights and Ideas
- Transformers are "Blacker" Boxes: Agreement between explanation methods is substantially lower for DistilBERT than for LSTMs. In Transformers, information mixes across layers so thoroughly that pinpointing a single token's "importance" becomes mathematically inconsistent across different XAI methods.
- The "Ideal" Explanation is a Myth: The fundamental flaw in "agreement as evaluation" is the assumption that there is one single "ground truth" ranking of word importance. In reality, a model might have multiple valid reasoning paths, or different XAI methods might simply be capturing different "slices" of model behavior.
- Visuals are Deceiving: The authors show that two heatmaps can look very similar to a human (e.g., both highlighting the word "technically" as important), yet have a near-zero rank correlation because the rankings of all the other words in the sentence are completely different.
- Math-Based Clusters: The only time methods agreed was when they shared underlying math (e.g., Integrated Gradients and Grad-SHAP). This suggests agreement is a function of the algorithm's design, not the model's actual "truth."
Key Results
- Low Correlation: For the DistilBERT model, the correlation between attention and attribution methods was consistently weak (often below 0.2 Kendall-τ).
- Attribution Disagreement: Even when comparing two non-attention methods (like LIME vs. IG), the correlation was surprisingly low in Transformers.
- Task Sensitivity: Agreement was higher for simple sentiment analysis (SST-2) but plummeted for complex pair-sequence tasks (MNLI/SNLI), where the relationship between words is more nuanced.
- The Verdict: Because even "gold standard" attribution methods don't agree with each other, using them to validate or invalidate attention as an explanation is logically unsound.
Limitations and Future Work
- Instance Sampling: Due to the high computational cost of methods like SHAP and LIME, the authors limited their evaluation to 500 instances per test set.
- Implementation Specifics: Results are tied to the specific implementations provided by the Captum library; different hyperparameters for LIME or IG might yield different results.
- Future Direction: The authors suggest moving toward human-in-the-loop processes. Instead of looking for mathematical agreement, we should ask if the explanations align with human intuition for specific use cases.
- Toy Data: They plan to develop "toy" datasets where a single, mathematically certain "correct" ranking exists to see if any XAI method is actually capable of recovering it.
Summary updated