A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing

Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, Ana Lucic

Abstract

There has been significant debate in the NLP community about whether or not attention weights can be used as an explanation—a mechanism for interpreting how important each input token is for a particular prediction. The validity of "attention as explanation" has so far been evaluated by computing the rank correlation between attention-based explanations and existing feature attribution explanations using LSTM-based models. In our work, we (i) compare the rank correlation between five more recent feature attribution methods and two attention-based methods, on two types of NLP tasks, and (ii) extend this analysis to also include transformer-based models. We find that attention-based explanations do not correlate strongly with any recent feature attribution methods, regardless of the model or task. Furthermore, we find that none of the tested explanations correlate strongly with one another for the transformer-based model, leading us to question the underlying assumption that we should measure the validity of attention-based explanations based on how well they correlate with existing feature attribution explanation methods. After conducting experiments on five datasets using two different models, we argue that the community should stop using rank correlation as an evaluation metric for attention-based explanations. We suggest that researchers and practitioners should instead test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand.

Introduction

In Natural Language Processing (NLP), understanding why a model makes a specific prediction is as crucial as the prediction itself. This need has birthed the field of Explainable AI (XAI), where researchers seek to identify which input tokens most heavily influence a model's output. Two primary approaches have emerged: attention weights, which represent the model's internal focus during processing, and feature attribution methods, which use mathematical techniques to calculate a word's contribution post-hoc.

However, a central controversy persists: Is attention a faithful explanation? Previous research suggested it was not, primarily because attention rankings often fail to correlate with gradient-based rankings. This established a paradigm known as "agreement as evaluation"—the assumption that if two explanation methods disagree, one must be unfaithful. This paper challenges that assumption by investigating whether the "gold standard" attribution methods even agree with each other.

Experimental Framework

To test the validity of the agreement paradigm, the researchers conducted a large-scale empirical study across different architectures and tasks.

Models and Methods

The study compared two distinct architectures:

  1. Bi-LSTM: A traditional recurrent model using additive attention.
  2. DistilBERT: A modern, efficient Transformer-based model.

These models were evaluated against two attention-based methods (Raw Attention for LSTMs and Attention Rollout for Transformers) and five state-of-the-art feature attribution methods:

  • LIME: A local surrogate model that perturbs inputs to observe output changes.
  • Integrated Gradients (IG): A gradient-based approach that attributes the prediction to input features.
  • DeepLIFT: A method that attributes changes in output to input features relative to a reference input.
  • Grad-SHAP & Deep-SHAP: Variations of Shapley values rooted in game theory.

The primary metric for comparison was Kendall-τ (tau) rank correlation, which quantifies the similarity between how two different methods rank the importance of tokens within a sentence.

Datasets and Tasks

The evaluation spanned five datasets representing two core NLP challenges:

  • Single-sequence (Sentiment Analysis): SST-2 and IMDb movie reviews.
  • Pair-sequence (Inference/Similarity): MNLI, SNLI (textual entailment), and Quora Question Pairs (QQP).

Key Insights

The Transformer Complexity Gap

One of the most significant findings is that agreement between explanation methods is substantially lower for DistilBERT than for LSTMs. In Transformers, information is mixed across multiple layers and heads so thoroughly that pinpointing a single token's "importance" becomes mathematically inconsistent across different XAI methods. This suggests that as models become more complex, the notion of a single, stable explanation becomes increasingly elusive.

The Myth of the Ground Truth

The fundamental flaw in using rank correlation as an evaluation metric is the assumption that a single "ground truth" ranking of word importance exists. In reality, a model may utilize multiple valid reasoning paths, or different XAI methods may simply be capturing different facets of model behavior. For instance, two heatmaps might visually highlight the same key word to a human observer, yet have a near-zero rank correlation because the rankings of the remaining, less-important words differ wildly.

Algorithmic Clustering

Agreement was found to be a function of algorithmic design rather than model "truth." Methods only tended to agree when they shared underlying mathematical principles (e.g., Integrated Gradients and Grad-SHAP). This implies that using one method to validate another often results in a circular logic where we are merely measuring the similarity of the algorithms' math, not their faithfulness to the model's internal logic.

Results and Discussion

  • Weak Correlations: For DistilBERT, the correlation between attention and attribution methods was consistently weak, often falling below 0.2 Kendall-τ.
  • Attribution Disagreement: Even when comparing two non-attention methods (e.g., LIME vs. IG), the correlation remained surprisingly low in Transformer models.
  • Task Sensitivity: While agreement was slightly higher for simple sentiment analysis (SST-2), it plummeted for complex pair-sequence tasks (MNLI/SNLI), where the relationships between words are more nuanced.

Conclusion

Because even the supposed "gold standard" attribution methods do not agree with one another, using them to validate or invalidate attention as an explanation is logically unsound. The research community should move away from rank correlation as a definitive metric for XAI faithfulness. Instead, the authors recommend a shift toward human-in-the-loop processes, where the value of an explanation is determined by its utility and alignment with human intuition for specific, real-world use cases. Future work will involve developing "toy" datasets with mathematically certain importance rankings to further test the recovery capabilities of these XAI methods.

Summary updated

Abstract

There has been significant debate in the NLP community about whether or not attention weights can be used as an explanation - a mechanism for interpreting how important each input token is for a particular prediction. The validity of "attention as explanation" has so far been evaluated by computing the rank correlation between attention-based explanations and existing feature attribution explanations using LSTM-based models. In our work, we (i) compare the rank correlation between five more recent feature attribution methods and two attention-based methods, on two types of NLP tasks, and (ii) extend this analysis to also include transformer-based models. We find that attention-based explanations do not correlate strongly with any recent feature attribution methods, regardless of the model or task. Furthermore, we find that none of the tested explanations correlate strongly with one another for the transformer-based model, leading us to question the underlying assumption that we should measure the validity of attention-based explanations based on how well they correlate with existing feature attribution explanation methods. After conducting experiments on five datasets using two different models, we argue that the community should stop using rank correlation as an evaluation metric for attention-based explanations. We suggest that researchers and practitioners should instead test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand.
BibTeX
@inproceedings{neely2022song,
  author = {Michael Neely and Stefan F. Schouten and Maurits J. R. Bleeker and Ana Lucic},
  title = {{A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing}},
  booktitle = {HHAI},
  year = {2022},
  doi = {10.3233/faia220190},
}
Figure