When we deploy a complex neural network, we often want to know why it made a specific prediction. To find out, practitioners turn to post-hoc feature attribution methods like LIME, SHAP, Integrated Gradients, or vanilla saliency maps. These tools promise to look inside the black box and assign an "importance score" to each input feature.
However, if you apply multiple attribution methods to the exact same model for the exact same prediction, a troubling pattern emerges: the methods frequently give completely different answers. One tool might claim a patient's age was the decisive factor in a medical diagnosis, while another claims it was their blood pressure.
This phenomenon is known as the disagreement problem. It is not a rare edge case; it is a pervasive structural issue across vision, language, and tabular models. Empirical evaluations, such as those by Alex Rodriguez et al. in When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances, demonstrate that feature importance rankings can exhibit near-zero correlation across different explainers. This lack of consensus creates a crisis of trust, especially in high-stakes domains like healthcare or law.
To understand why these explanations contradict one another, we have to look at the underlying mechanics of how attribution methods probe a model's decision space.
The "No Free Lunch" of Local Function Approximation
The root cause of the disagreement problem is that different attribution methods are actually optimizing for entirely different mathematical objectives. They are not extracting a single, objective "ground truth" from the model; instead, they are creating simplified approximations of the model's behavior from different perspectives.
In Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations, Tessa Han et al. unified eight popular post-hoc explanation methods under a single mathematical framework called Local Function Approximation (LFA). The authors proved a "No Free Lunch" theorem for explainability: no single explanation method can simultaneously satisfy all desirable properties, such as local fidelity (perfectly matching the model nearby) and global awareness.
Similarly, Fiona Katharina Ewald demonstrated in A Guide to Feature Importance Methods for Scientific Inference that different feature importance metrics map to fundamentally distinct types of statistical associations within the data-generating process. For instance, some methods measure how much a feature uniquely contributes to a prediction, while others capture its shared contribution alongside correlated features.
When methods disagree, it is often because they are answering different questions. As Zifan Wang et al. articulated in Interpreting Interpretations: Organizing Attribution Methods by Criteria, some explainers excel at finding necessary features (features without which the prediction changes), while others find sufficient features (features that can drive the prediction entirely on their own).
The Arbitrariness of Baselines and Perturbations
Many popular attribution methods evaluate feature importance by observing how the model's output changes when inputs are removed or altered. But how do you "remove" a feature from a neural network that expects a fixed-size input vector?
Methods like SHAP and Integrated Gradients solve this by replacing the feature with a "baseline" or reference value (such as a black pixel, a zero vector, or an average value). However, the choice of this baseline is highly subjective and profoundly shapes the resulting explanation.
In A study on the Interpretability of Neural Retrieval Models using DeepSHAP, Zeon Trevor Fernando et al. showed that explanations are highly sensitive to what is defined as a "neutral" reference input. This vulnerability was explored deeply by Marvin N. Wright et al. in Toward Understanding the Disagreement Problem in Neural Network Feature Attribution. Through controlled simulations, the authors proved that the disagreement problem is largely driven by data preprocessing choices and baseline values rather than inherent flaws in the attribution algorithms themselves. If two methods use slightly different assumptions about what constitutes "absence," their importance rankings will inevitably diverge.
Furthermore, gradient-based methods (which look at the local slope of the decision boundary) and perturbation-based methods (which sample the model's output across a wider neighborhood) probe different parts of the function space. A model's landscape might be flat locally but drop off steeply a short distance away, causing gradient and perturbation methods to disagree even on the sign of a feature's influence, as noted by Gianluigi Lopardo et al. in Attention Meets Post-hoc Interpretability: A Mathematical Perspective.
Feature Interactions and the Coalition Conflict
In real-world data, features rarely act in isolation; they interact in complex, non-linear ways. When a neural network learns these interactions, distributing credit to individual features becomes mathematically ambiguous.
Foutse Khomh argued in Tackling the XAI Disagreement Problem with Regional Explanations that feature interactions are the primary catalyst for the disagreement problem. When a model relies on non-additive logic (e.g., an AND gate requiring both Feature A and Feature B to be present), standard additive attribution methods struggle to allocate weight fairly.
This creates what Xinhao Zheng et al. termed an attribution conflict in Towards Attributions of Input Variables in a Coalition. The authors showed a significant mathematical discrepancy between the importance assigned to a group of features (a coalition) and the sum of the importance values assigned to its individual members. Because different explainers use different heuristics to break down these coalitions, they arrive at wildly divergent individual feature rankings.
Resolving Disagreement
Rather than searching for a single "perfect" explainer that does not exist, recent research focuses on mitigating disagreement by changing how we aggregate and bound explanations.
1. Regional Partitioning
Instead of forcing a global or purely local explanation, researchers are finding that explainers align much better within specific sub-regions of the data space. Julia Herbinger et al. introduced this concept via the GRANITE framework in GRANITE: A Generalized Regional Framework for Identifying Agreement in Feature-Based Explanations. By recursively partitioning the feature space into regions where feature interactions are stable, they isolate the specific contexts where different attribution methods actually reach a consensus.
2. Dynamic Thresholding
Often, explainers agree on the general area of importance but disagree on the exact ranking of lower-tier features. In Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods, Jonathan Kamp et al. demonstrated that evaluating a fixed number of "Top-$k$" features artificially inflates disagreement. By switching to a dynamic threshold that identifies "peaks" in attribution scores, cross-method consensus improves dramatically.
3. Representation Disentanglement
We can also train models to be inherently easier to explain. Josip Jukic showed in Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods that applying regularization techniques to disentangle hidden representations makes the model's internal logic cleaner. When the underlying representations are less tangled, different saliency methods naturally converge on the same explanations without sacrificing the model's predictive accuracy.
Key Foundational Work
A study on the Interpretability of Neural Retrieval Models using DeepSHAP (2019) — contributed 3 concept elements to this direction.