Abstract
Methods for interpreting machine learning black-box models increase the outcomes' transparency and in turn generates insight into the reliability and fairness of the algorithms. However, the interpretations themselves could contain significant uncertainty that undermines the trust in the outcomes and raises concern about the model's reliability. Focusing on the method "Local Interpretable Model-agnostic Explanations" (LIME), we demonstrate the presence of two sources of uncertainty, namely the randomness in its sampling procedure and the variation of interpretation quality across different input data points. Such uncertainty is present even in models with high training and test accuracy. We apply LIME to synthetic data and two public data sets, text classification in 20 Newsgroup and recidivism risk-scoring in COMPAS, to support our argument.
Introduction
As machine learning models are increasingly deployed in high-stakes domains like recidivism prediction and medical diagnosis, interpretability has transitioned from a luxury to a requirement. Post-hoc explanation methods, most notably Local Interpretable Model-Agnostic Explanations (LIME), are frequently employed to demystify the decisions of "black-box" models. LIME operates by perturbing input data points and fitting a simple, interpretable linear model to the resulting local neighborhood.
However, a critical meta-problem persists: if the explanation itself is unreliable or unstable, it cannot foster genuine trust. If repeated applications of LIME to the same data point yield divergent explanations, the perceived transparency becomes an illusion. This research quantifies this uncertainty, identifying its sources and determining the conditions under which an explanation can be considered trustworthy.
Methodology: Decomposing Uncertainty
The authors identify three primary dimensions of uncertainty within the LIME framework:
- Sampling Variance: LIME's reliance on random sampling in the vicinity of a data point means that different iterations can produce different sets of important features.
- Proximity Sensitivity: LIME utilizes a kernel to weight neighboring points. The choice of "sampling proximity" (the neighborhood radius) significantly influences whether the explanation captures local nuances or reverts to global model trends.
- Explained Model Credibility: This assesses whether the features identified by LIME align with domain knowledge and the real-world context of the data.
To rigorously test these factors, the authors adopted a "Mimic Model" approach, training interpretable models (such as Random Forests) on complex data and evaluating LIME's ability to recover their known internal logic.
Experimental Setup and Datasets
The study utilizes three distinct datasets to probe LIME's performance:
- Synthetic Tree-Generated Data: A controlled environment where data is partitioned by a decision tree. Each leaf node has a specific set of active linear weights, allowing for a precise ground truth of local vs. global feature importance. Synthetic data generation using a decision tree to define local feature importance. Source, p. 2
- 20 Newsgroups: A standard text classification dataset. The focus was on categories like "Atheism vs. Christianity" to determine if LIME identifies meaningful semantic markers or arbitrary noise.
- COMPAS Recidivism Risk-Scoring: A real-world dataset used for predicting re-offending likelihood. This was used to check if LIME consistently identifies sensitive features like race and age.
Key Insights: The Local-Global Tension
A major finding is the "Global Bias" inherent in default LIME settings. By default, LIME often prioritizes features that are globally important to the model, potentially overlooking the specific local logic relevant to an individual data point.
The authors discovered that shrinking the sampling radius (proximity) by an order of magnitude allowed LIME to successfully identify local features in synthetic tests. However, the lack of a standardized method for selecting the "correct" radius remains a significant hurdle for practitioners.
Results and Analysis
The experiments revealed varying levels of stability across different domains:
- Synthetic Recovery: In an 8-feature model, LIME initially failed to capture the local importance of specific feature subsets. Adjusting the sampling proximity was necessary to shift selection probabilities from near-zero to significant levels.
- Text Variance and Nonsense Explanations: In text classification, LIME's performance was highly inconsistent. While it was 100% stable for some documents (e.g., consistently picking "crypto"), for others it highlighted uninformative artifacts like "re", "not", or "being" with high frequency. Unstable and uninformative word features identified by LIME in text classification. Source, p. 4
- Stability in COMPAS: Conversely, LIME demonstrated high stability on the COMPAS dataset, consistently flagging "race" and "age." This suggests that when model biases are systemic, LIME is more likely to produce a statistically stable (though ethically concerning) explanation.
Discussion and Future Work
The research underscores that "trust" in an explanation is not binary. An explanation can be statistically stable (low variance) yet contextually useless (low credibility).
Currently, the burden of verifying explanation stability falls on the user, who must run LIME multiple times and manually inspect results. Future work should focus on:
- Principled Parameter Selection: Developing automated methods for choosing the optimal sampling proximity (sigma).
- Confidence Intervals: Integrating uncertainty metrics directly into explanation tools to provide users with a measure of "explanation confidence."
- Broader Scope: Extending this analysis to other local explanation methods, such as SHAP, to determine if these uncertainty patterns are universal across the field of interpretable machine learning.
Summary updated