Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods

Josip Jukic, Martin Tutek, Jan Snajder

Introduction

Saliency methods, such as Integrated Gradients and DeepLIFT, are widely used to interpret the predictions of "black box" neural networks by assigning importance scores to input features. However, these methods frequently suffer from a "disagreement problem," where different techniques provide conflicting explanations for the same model and input. This inconsistency creates a crisis of faith in Explainable AI (XAI), as it remains unclear which method a human should trust. This paper investigates the root causes of these disagreements and proposes both a metric shift and architectural regularization to harmonize model explanations.

The Metric Problem: Kendall-$τ$ vs. Pearson-$r$

Previous research into saliency agreement has largely relied on Kendall-$τ$, a rank-based correlation metric. The authors argue that this metric is fundamentally flawed for this purpose because it is overly sensitive to tiny, noisy fluctuations in the "long tail" of unimportant tokens. In many NLP tasks, only a few words are truly decisive; the exact ordinal ranking of the remaining irrelevant words should not significantly impact the agreement score.

![A toy example demonstrating how small perturbations in irrelevant tokens can drastically lower Kendall-τ] while Pearson-r remains stable.](/figures/art_ca1bbc74679a48839beb9ef305c3b581)

By switching to Pearson-$r$ (linear correlation), which focuses on the relative magnitude of importance scores, researchers can obtain a more robust measure of agreement. As shown in the toy example above, a minor shuffle of low-importance scores can tank a Kendall score to -0.43, while the Pearson correlation remains a near-perfect 0.99, reflecting the shared identification of the primary feature.

Representation Entanglement and Regularization

The authors hypothesize that disagreements often stem from "representation entanglement" in the model's hidden layers. When hidden states are too similar to one another, gradient-based saliency methods struggle to trace importance back to specific inputs, resulting in noisy and inconsistent paths. To combat this, the paper introduces two regularization techniques:

  1. CONICITY (C): This forces hidden states to spread out in the vector space, increasing the angle between them to ensure distinct representations.
  2. TYING (T): This minimizes the distance between a hidden state and its corresponding input word embedding, forcing the model to maintain a more direct relationship between tokens and their internal representations.

Insights from Dataset Cartography

Using "Dataset Cartography," the authors categorized training data into Easy-to-learn, Hard-to-learn, and Ambiguous instances. A counterintuitive finding emerged: saliency methods agree the least on easy-to-learn instances. This is attributed to the Rashomon effect, where multiple "correct-looking" explanations exist for simple decisions. Conversely, agreement is highest for ambiguous instances where the model is "on the fence," forcing all methods to focus on the specific sharp features that push the model across the decision boundary.

Key Results

The proposed regularization techniques, particularly TYING, significantly improved agreement across various models (Bi-LSTM with attention and DistilBERT) and datasets (SUBJ, SST, TREC, IMDB).

Table showing the significant boost in agreement metrics (τ_k and r_p) when using Tying (T) and Conicity (C) regularization compared to the Base (B) model.

As seen in the results, for the JWA model on the SUBJ dataset, Pearson agreement ($r_p$) jumped from 0.66 to 0.88 when applying TYING. Crucially, these improvements in interpretability did not come at the cost of model performance; the regularized models maintained $F_1$ scores comparable to their unregularized counterparts.

Conclusion

By shifting to magnitude-sensitive metrics like Pearson-$r$ and employing regularization to disentangle hidden representations, we can produce models that are not only accurate but also more consistently interpretable. While the lack of "ground truth" for importance remains a challenge in XAI, reducing arbitrary disagreements between methods is a vital step toward building trustworthy AI systems.

Summary updated

Abstract

A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement -- if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for the use of alternative diagnostic methods. In our work, we demonstrate that rank correlation is not a good fit for evaluating agreement and argue that Pearson-$r$ is a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods.
BibTeX
@inproceedings{jukic2023easy,
  author = {Josip Jukic and Martin Tutek and Jan Snajder},
  title = {{Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=pz4zQAcySU},
}
Figure