A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing

Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, Ana Lucic

TL;DR

This paper investigates whether comparing different AI explanation methods—specifically attention weights versus feature attribution—is a valid way to evaluate "faithfulness." The authors find that these methods rarely agree on which words are important, especially in Transformer models, leading them to argue that the community should abandon rank correlation as a standard evaluation metric for Explainable AI (XAI).

What Problem Does This Paper Address?

In Natural Language Processing (NLP), we often want to know why a model made a specific prediction. Two popular ways to do this are looking at attention weights (how much the model "focused" on a word) and feature attribution methods (mathematical techniques like LIME or SHAP that calculate a word's contribution).

A heated debate exists: Is attention actually an explanation? Previous influential work argued "no" because attention rankings don't correlate well with gradient-based rankings. This created a paradigm called "agreement as evaluation"—the idea that if two explanation methods don't agree, one must be unfaithful.

The authors of this paper question this paradigm. If two "trusted" attribution methods don't even agree with each other, how can we use one to debunk another?

Methods

The researchers conducted a large-scale empirical study to test the "agreement" hypothesis across different architectures and tasks.

  • Models: They compared a traditional Bi-LSTM (with additive attention) against a modern Transformer (DistilBERT).
  • Explanation Methods: They tested two attention-based methods (Raw Attention for LSTMs and Attention Rollout for Transformers) against five state-of-the-art feature attribution methods:
  • LIME: Perturbs inputs to see how the output changes.
  • Integrated Gradients (IG): A gradient-based approach.
  • DeepLIFT: Attributes changes in output to input features.
  • Grad-SHAP & Deep-SHAP: Two variations of Shapley values, which are rooted in game theory.
  • The Metric: They used Kendall-τ (tau) rank correlation, which measures how similarly two methods rank the importance of tokens in a sentence.

Datasets

The study used five diverse datasets covering two types of NLP tasks:

  1. Single-sequence (Sentiment Analysis): SST-2 and IMDb movie reviews.
  2. Pair-sequence (Inference/Similarity): MNLI, SNLI (textual entailment), and Quora Question Pairs (identifying duplicate questions).

Key Insights and Ideas

  • Transformers are "Blacker" Boxes: Agreement between explanation methods is substantially lower for DistilBERT than for LSTMs. In Transformers, information mixes across layers so thoroughly that pinpointing a single token's "importance" becomes mathematically inconsistent across different XAI methods.
  • The "Ideal" Explanation is a Myth: The fundamental flaw in "agreement as evaluation" is the assumption that there is one single "ground truth" ranking of word importance. In reality, a model might have multiple valid reasoning paths, or different XAI methods might simply be capturing different "slices" of model behavior.
  • Visuals are Deceiving: The authors show that two heatmaps can look very similar to a human (e.g., both highlighting the word "technically" as important), yet have a near-zero rank correlation because the rankings of all the other words in the sentence are completely different.
  • Math-Based Clusters: The only time methods agreed was when they shared underlying math (e.g., Integrated Gradients and Grad-SHAP). This suggests agreement is a function of the algorithm's design, not the model's actual "truth."

Key Results

  • Low Correlation: For the DistilBERT model, the correlation between attention and attribution methods was consistently weak (often below 0.2 Kendall-τ).
  • Attribution Disagreement: Even when comparing two non-attention methods (like LIME vs. IG), the correlation was surprisingly low in Transformers.
  • Task Sensitivity: Agreement was higher for simple sentiment analysis (SST-2) but plummeted for complex pair-sequence tasks (MNLI/SNLI), where the relationship between words is more nuanced.
  • The Verdict: Because even "gold standard" attribution methods don't agree with each other, using them to validate or invalidate attention as an explanation is logically unsound.

Limitations and Future Work

  • Instance Sampling: Due to the high computational cost of methods like SHAP and LIME, the authors limited their evaluation to 500 instances per test set.
  • Implementation Specifics: Results are tied to the specific implementations provided by the Captum library; different hyperparameters for LIME or IG might yield different results.
  • Future Direction: The authors suggest moving toward human-in-the-loop processes. Instead of looking for mathematical agreement, we should ask if the explanations align with human intuition for specific use cases.
  • Toy Data: They plan to develop "toy" datasets where a single, mathematically certain "correct" ranking exists to see if any XAI method is actually capable of recovering it.

Summary updated

Abstract

There has been significant debate in the NLP community about whether or not attention weights can be used as an explanation - a mechanism for interpreting how important each input token is for a particular prediction. The validity of "attention as explanation" has so far been evaluated by computing the rank correlation between attention-based explanations and existing feature attribution explanations using LSTM-based models. In our work, we (i) compare the rank correlation between five more recent feature attribution methods and two attention-based methods, on two types of NLP tasks, and (ii) extend this analysis to also include transformer-based models. We find that attention-based explanations do not correlate strongly with any recent feature attribution methods, regardless of the model or task. Furthermore, we find that none of the tested explanations correlate strongly with one another for the transformer-based model, leading us to question the underlying assumption that we should measure the validity of attention-based explanations based on how well they correlate with existing feature attribution explanation methods. After conducting experiments on five datasets using two different models, we argue that the community should stop using rank correlation as an evaluation metric for attention-based explanations. We suggest that researchers and practitioners should instead test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand.
BibTeX
@inproceedings{neely2022song,
  author = {Michael Neely and Stefan F. Schouten and Maurits J. R. Bleeker and Ana Lucic},
  title = {{A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing}},
  booktitle = {HHAI},
  year = {2022},
  doi = {10.3233/faia220190},
}

Loading related content...

Figure