Abstract
As machine learning models grow in complexity, their "black box" nature poses a significant barrier to trust and deployment. This paper introduces LIME (Local Interpretable Model-agnostic Explanations), a framework that explains the predictions of any classifier by approximating it locally with an interpretable model. Furthermore, we present SP-LIME, a method for selecting a representative set of instances to provide a global understanding of the model's behavior. Our experiments demonstrate that LIME helps users identify untrustworthy models and improve them through feature engineering.
Introduction: The Trust Gap in Machine Learning
While high accuracy on a test set is often the primary goal in model development, it can be a misleading metric for real-world reliability. Models may achieve high performance by exploiting "data leakage" or "artifacts"—irrelevant patterns in the training data that do not generalize. If practitioners cannot understand the reasoning behind a prediction, they are unlikely to trust or deploy the model. This research aims to make model behavior transparent and human-understandable, bridging the gap between complex algorithmic outputs and human intuition.
Methodology: LIME and SP-LIME
The authors propose two core techniques to address interpretability:
1. LIME (Local Interpretable Model-agnostic Explanations)
LIME is designed to be model-agnostic, meaning it can explain any classifier (e.g., neural networks, SVMs, or Random Forests) without accessing its internal architecture. It achieves local fidelity by focusing on a single prediction. The process involves "perturbing" the input—such as hiding words in a document—and observing how the model's output changes. A simple, interpretable model (like a linear model) is then trained on these perturbed samples, weighted by their proximity to the original input. This local surrogate model provides an explanation that is faithful to the black box in that specific neighborhood.
2. SP-LIME (Submodular Pick)
To move from local to global trust, the authors introduce SP-LIME. This method uses a greedy optimization algorithm to select a diverse and non-redundant set of instances and their corresponding LIME explanations. This provides users with a representative "gallery" of the model's logic across the entire dataset, preventing them from being overwhelmed by a massive number of examples.
Key Insights: The "Right for the Wrong Reasons" Trap
A central finding of the paper is that models can be highly accurate yet fundamentally flawed. In a classification task distinguishing "Atheism" from "Christianity," a model achieved 94% accuracy. However, LIME revealed that the model was primarily relying on email headers (e.g., "Host:", "NNTP-Posting-Host:") rather than the actual content of the posts.
This insight highlights that interpretability is domain-specific. LIME maps complex features (like word embeddings) back to "interpretable representations" (like the presence of specific words) that humans can evaluate. By identifying these artifacts, users can perform "feature cleaning"—removing noisy or irrelevant words—to improve the model's generalization.
Experimental Results
The framework was evaluated on several NLP tasks, including the 20 Newsgroups dataset, politeness detection, and sentiment analysis. Key findings include:
- Efficiency: Explanations are typically generated in under three seconds, supporting interactive use.
- Model Comparison: User studies showed that LIME enabled participants to correctly identify which of two classifiers would generalize better, even when their training accuracies were nearly identical.
- Trust Building: Visual artifacts, such as highlighted text and feature importance charts, significantly enhanced a user's ability to diagnose and rectify untrustworthy models.
Conclusion and Future Directions
While LIME and SP-LIME provide powerful tools for interpretability, challenges remain. There is an inherent trade-off between the complexity of an explanation and its interpretability for humans. Furthermore, while SP-LIME offers a global view, it remains a collection of local snapshots. Future work may explore more complex feature interactions and deeper global logic within massive neural networks.
Summary updated