"Why Should I Trust You?": Explaining the Predictions of Any Classifier

Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin

Abstract

As machine learning models grow in complexity, their "black box" nature poses a significant barrier to trust and deployment. This paper introduces LIME (Local Interpretable Model-agnostic Explanations), a framework that explains the predictions of any classifier by approximating it locally with an interpretable model. Furthermore, we present SP-LIME, a method for selecting a representative set of instances to provide a global understanding of the model's behavior. Our experiments demonstrate that LIME helps users identify untrustworthy models and improve them through feature engineering.

Introduction: The Trust Gap in Machine Learning

While high accuracy on a test set is often the primary goal in model development, it can be a misleading metric for real-world reliability. Models may achieve high performance by exploiting "data leakage" or "artifacts"—irrelevant patterns in the training data that do not generalize. If practitioners cannot understand the reasoning behind a prediction, they are unlikely to trust or deploy the model. This research aims to make model behavior transparent and human-understandable, bridging the gap between complex algorithmic outputs and human intuition.

Methodology: LIME and SP-LIME

The authors propose two core techniques to address interpretability:

1. LIME (Local Interpretable Model-agnostic Explanations)

LIME is designed to be model-agnostic, meaning it can explain any classifier (e.g., neural networks, SVMs, or Random Forests) without accessing its internal architecture. It achieves local fidelity by focusing on a single prediction. The process involves "perturbing" the input—such as hiding words in a document—and observing how the model's output changes. A simple, interpretable model (like a linear model) is then trained on these perturbed samples, weighted by their proximity to the original input. This local surrogate model provides an explanation that is faithful to the black box in that specific neighborhood.

Figure 4: Components of a LIME explanation, including prediction probabilities, local feature importance, and highlighted text.

2. SP-LIME (Submodular Pick)

To move from local to global trust, the authors introduce SP-LIME. This method uses a greedy optimization algorithm to select a diverse and non-redundant set of instances and their corresponding LIME explanations. This provides users with a representative "gallery" of the model's logic across the entire dataset, preventing them from being overwhelmed by a massive number of examples.

Key Insights: The "Right for the Wrong Reasons" Trap

A central finding of the paper is that models can be highly accurate yet fundamentally flawed. In a classification task distinguishing "Atheism" from "Christianity," a model achieved 94% accuracy. However, LIME revealed that the model was primarily relying on email headers (e.g., "Host:", "NNTP-Posting-Host:") rather than the actual content of the posts.

Figure 1: LIME interface showing how a model relies on non-content features like headers to distinguish between Atheism and Christianity.

This insight highlights that interpretability is domain-specific. LIME maps complex features (like word embeddings) back to "interpretable representations" (like the presence of specific words) that humans can evaluate. By identifying these artifacts, users can perform "feature cleaning"—removing noisy or irrelevant words—to improve the model's generalization.

Experimental Results

The framework was evaluated on several NLP tasks, including the 20 Newsgroups dataset, politeness detection, and sentiment analysis. Key findings include:

  • Efficiency: Explanations are typically generated in under three seconds, supporting interactive use.
  • Model Comparison: User studies showed that LIME enabled participants to correctly identify which of two classifiers would generalize better, even when their training accuracies were nearly identical.
  • Trust Building: Visual artifacts, such as highlighted text and feature importance charts, significantly enhanced a user's ability to diagnose and rectify untrustworthy models.

Conclusion and Future Directions

While LIME and SP-LIME provide powerful tools for interpretability, challenges remain. There is an inherent trade-off between the complexity of an explanation and its interpretability for humans. Furthermore, while SP-LIME offers a global view, it remains a collection of local snapshots. Future work may explore more complex feature interactions and deeper global logic within massive neural networks.

Summary updated

Abstract

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.
BibTeX
@inproceedings{ribeiro2016why,
  author = {Marco Túlio Ribeiro and Sameer Singh and Carlos Guestrin},
  title = {{"Why Should I Trust You?": Explaining the Predictions of Any Classifier}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-3020},
}
Figure