Introduction
As machine learning models grow in complexity, the need for transparency has led to the widespread adoption of interpretability methods like LIME and SHAP. These tools aim to provide human-understandable explanations for individual model predictions. However, a critical requirement for any meaningful explanation is robustness: similar inputs should yield similar explanations. This paper investigates the stability of popular interpretability techniques and reveals a concerning level of fragility.
The Problem: Brittle Explanations
Most current interpretability methods are prediction-based, explaining a specific output for a specific input. While we expect a model's prediction to remain stable under minor input perturbations, we have not held explanations to the same standard. If adding imperceptible noise to an input causes its explanation to change drastically—even if the model's classification remains the same—the explanation is "brittle." Such instability undermines trust and makes these tools unreliable for high-stakes applications like safety audits or legal compliance.
Measuring Robustness via Lipschitz Continuity
The authors propose a formal framework to quantify explanation stability using Local Lipschitz Continuity. They define a "Lipschitz estimate" for an explanation function, which measures the maximum ratio of the change in the explanation to the change in the input within a small neighborhood. A high Lipschitz value indicates an unstable, non-robust explanation.
To evaluate this, the authors compared several classes of methods:
- Perturbation-based (Black-box): LIME, SHAP, and Occlusion sensitivity.
- Gradient-based (White-box): Saliency maps, Integrated Gradients, and ε-LRP.
As shown in the figure above, both SHAP and LIME attempt to approximate the model's local behavior. However, their reliance on sampling and local surrogates can lead to inconsistencies, especially in regions where the decision boundary is complex.
Key Findings: The Fragility of Black-Box Methods
The study reveals that LIME is particularly fragile. Because it relies on random sampling and sparse "super-pixels," it is highly sensitive to small perturbations. In quantitative tests across various datasets (UCI benchmarks, MNIST, ImageNet), LIME consistently showed the highest Lipschitz estimates, indicating the lowest robustness.
In contrast, gradient-based methods—which have access to the model's internal weights—tend to be more stable. Among these, Integrated Gradients performed the best, though it still exhibited some instability. The authors argue that black-box methods often "hallucinate" feature importance based on noisy samples, whereas gradient-based methods are more grounded in the model's actual logic.
Real-World Impact: The COMPAS Dataset
The implications of these findings are most evident in sensitive domains. Using the COMPAS dataset (criminal recidivism risk), the authors demonstrated how nearly identical inputs can lead to wildly different explanations.
In the example above, minor variations in input values (such as a slight change in the number of prior offenses) result in different features being highlighted as primary drivers of the prediction. For a defendant or a judge, this lack of consistency makes the explanation essentially meaningless.
Conclusion
Robustness is not merely a "nice-to-have" feature; it is a fundamental requirement for interpretability. If an explanation is not stable in a small neighborhood, it cannot serve as a reliable proxy for a complex model. This research highlights the need for new interpretability methods that are robust by construction, potentially by incorporating Lipschitz constraints directly into the training process. Until then, practitioners should exercise caution when relying on brittle explanations for critical decision-making.
Summary updated