Abstract
A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement -- if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for the use of alternative diagnostic methods. In our work, we demonstrate that rank correlation is not a good fit for evaluating agreement and argue that Pearson-$r$ is a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods.
Introduction: The Disagreement Crisis
As neural NLP models are increasingly deployed in high-stakes domains like law and medicine, the need for faithful explanations of their internal logic has become paramount. Saliency methods, such as Integrated Gradients or Deep-SHAP, are the primary tools for this task, assigning importance scores to individual input tokens to explain why a model made a specific prediction.
However, these methods often suffer from a "crisis of faith": they frequently disagree on which words are most influential for a given prediction. As illustrated above, two different methods can produce markedly different importance rankings for the same sentence. This lack of consensus suggests that current interpretability methods may not be capturing the model's true reasoning process, or that the models themselves are represented in a way that makes consistent explanation difficult.
Methodology: Metrics and Regularization
This research addresses the disagreement problem through three distinct lenses:
- Metric Reform: The authors argue that the standard metric for agreement, Kendall-$ au$ (a rank-based correlation), is overly sensitive to noise. Small, irrelevant changes in the scores of unimportant words can tank the rank correlation. They advocate for Pearson-$r$, which looks at the linear relationship between scores and is more robust to these minor perturbations.
- Representation Regularization: They hypothesize that "representation entanglement"—where hidden states are too similar—causes saliency methods to struggle. They apply two techniques: CONICITY, which spreads out hidden states in the representation space, and TYING, which forces hidden states to stay closer to their corresponding input word embeddings.
- Dataset Cartography: Using training dynamics (how quickly and consistently a model learns an instance), they categorize data into "easy-to-learn," "hard-to-learn," and "ambiguous" groups to see where disagreements are most prevalent.
Key Insights: The "Easy-to-Learn" Paradox
A counter-intuitive finding of the study is that saliency methods disagree the most on "easy" instances—those the model learns quickly and with high confidence. The authors link this to the Rashomon effect: for an easy example, there are many different combinations of words that could lead to a correct, high-confidence prediction. In these "smooth" areas of the representation space, saliency methods struggle to find a single, definitive explanation.
As shown in Figure 2, base models (red dots) exhibit low agreement on high-confidence instances. Conversely, agreement is higher for "ambiguous" instances. By applying TYING regularization (blue crosses), the authors essentially "sharpen" the model's internal logic, shifting the distribution toward significantly higher agreement across all categories.
Key Results and Performance
- Pearson is Superior: Using Pearson-$r$ consistently yielded higher agreement scores than Kendall-$ au$, suggesting that previous research may have over-emphasized disagreements caused by insignificant rank swaps in the "tail" of unimportant words.
- Significant Agreement Gains: Regularization (especially TYING) significantly improved agreement across almost all datasets (IMDB, SST-2, SUBJ, TREC) and models (JWA, DistilBERT). For example, on the SUBJ dataset with the JWA model, agreement jumped from 0.66 to 0.88 when using TYING.
- Performance Stability: Crucially, these improvements in interpretability did not come at a major cost to accuracy. Most regularized models performed within 3 F1 points of the unregularized baselines.
Conclusion
While agreement does not strictly guarantee truth, reducing systematic disagreements is a vital step toward reliable XAI. By adopting more appropriate metrics like Pearson-$r$ and applying targeted regularization to "untangle" representations, we can move closer to models that are not only accurate but also consistently interpretable. Future work will explore if these findings hold for more complex tasks like sequence labeling or machine translation.
Summary updated