"Why Should I Trust You?": Explaining the Predictions of Any Classifier

Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin

Abstract

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is fundamental to assessing trust, which is critical when taking action based on a prediction or choosing whether to deploy a new model. This work proposes LIME (Local Interpretable Model-agnostic Explanations), a novel technique that explains the predictions of any classifier in an interpretable and faithful manner by learning an interpretable model locally around the prediction. We also propose SP-LIME, a method to explain models by presenting representative individual predictions in a non-redundant way. We demonstrate the flexibility of these methods across text and image classification, showing their utility in identifying why a classifier should not be trusted and improving untrustworthy models.

The Problem: The Black Box and the Trust Gap

As machine learning models—particularly deep neural networks and complex ensembles—grow in sophistication, they increasingly function as "black boxes." While a model might exhibit high accuracy on a test set, the underlying logic for its specific predictions often remains opaque. This lack of transparency creates a significant barrier to trust.

The authors distinguish between two levels of trust:

  1. Trusting a prediction: Can a user rely on a specific output (e.g., a medical diagnosis) to take action?
  2. Trusting a model: Will the model perform reliably in real-world deployment where data distributions may shift?

Without interpretability, models often exploit "spurious correlations" or data leakage. A classic example is a model classifying a photo as a "wolf" simply because of snow in the background, rather than the features of the animal itself.

Methodology: How LIME Works

LIME is model-agnostic, meaning it treats any classifier as a black box without requiring access to its internal architecture. The core idea is to approximate the complex global model with a simpler, interpretable one in the immediate vicinity of a specific prediction.

The LIME explanation process: from black-box prediction to human-interpretable features.

The process involves four key steps:

  • Perturbation: The original input (e.g., a text document or image) is modified to create a series of "perturbed" samples by randomly hiding features (e.g., removing words or masking image patches).
  • Local Mapping: These samples are fed into the black-box model to observe how the output changes.
  • Simple Proxy: A simple, interpretable model (such as sparse linear regression) is fitted to these perturbed samples, weighted by their proximity to the original input.
  • Interpretation: The weights of this proxy model reveal which features were most influential for that specific prediction.

Local Fidelity vs. Global Interpretability

A model might be too complex to explain globally. However, LIME operates on the principle of local fidelity: even if a decision boundary is highly non-linear, it usually behaves linearly within a small enough neighborhood of a single instance.

LIME approximates a complex global decision boundary with a local linear model around a specific instance.

To provide a broader view of the model, the authors introduced SP-LIME (Submodular Pick). This algorithm uses submodular optimization to select a diverse, non-redundant set of instances to give users a representative overview of the model’s global behavior.

Key Insights and Results

The authors evaluated LIME across various domains, including sentiment analysis, newsgroup classification, and ImageNet.

  • Catching Data Leakage: LIME revealed that a high-accuracy newsgroup classifier was actually relying on metadata in email headers (e.g., the word "Posting") rather than the content of the messages.
  • Human Trust and Model Selection: In a "Husky vs. Wolf" experiment, LIME explanations reduced misplaced trust in a flawed model from 37% to 11%. Furthermore, human subjects using LIME were able to select the better-generalizing model 89% of the time.
  • Empowering Non-Experts: The study demonstrated that even users without machine learning expertise could use LIME to identify and remove problematic features, significantly improving real-world model performance.

Conclusion and Future Directions

LIME provides a powerful framework for opening the black box of machine learning. By focusing on local explanations and interpretable representations, it allows users to diagnose model failures and build trust. While the computational cost of explaining complex image networks can be high, and extremely non-linear local regions remain a challenge, LIME represents a major step toward transparent and accountable AI. Future work includes extending these techniques to speech and video data and exploring alternative explanation families like decision trees.

Summary updated

Abstract

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.
BibTeX
@inproceedings{ribeiro2016why,
  author = {Marco Túlio Ribeiro and Sameer Singh and Carlos Guestrin},
  title = {{"Why Should I Trust You?": Explaining the Predictions of Any Classifier}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-3020},
}
Figure