On the Relationship Between Interpretability and Explainability in Machine Learning

Benjamin Leblanc

TL;DR

This paper argues that interpretability and explainability in machine learning are not substitutes but rather complementary concepts. By analyzing the limitations of each approach individually, the authors demonstrate how combining them can lead to more robust, trustworthy, and insightful machine learning systems.

What Problem Does This Paper Address?

In high-stakes domains like healthcare and finance, understanding why a machine learning model makes a certain prediction is as crucial as the prediction itself. This has led to two main schools of thought: one focusing on making complex "black-box" models understandable through explainability techniques, and another advocating for inherently interpretable models that are simpler by design. However, this dichotomy has created a fragmented research landscape where the strengths of one approach are often ignored by the other. This paper addresses the problem of this artificial separation, arguing that a more holistic approach, leveraging both interpretability and explainability, is necessary to overcome their individual shortcomings and build more reliable AI.

Methods

This paper is a position paper that primarily uses a literature review and conceptual analysis to make its case. The authors:

  • Identify and discuss the principal drawbacks of explainability (e.g., explanations can be wrong, misleading, or require blind trust) and interpretability (e.g., interpretability alone doesn't reveal all information, can be subjective, and might make training harder).
  • Argue for complementarity: They propose that by combining interpretable models with explainability techniques, the limitations of each can be mitigated. For instance, an interpretable model can serve as a ground truth to verify explanations, and explainability can help overcome the subjectivity and information gaps of interpretability.
  • Propose a new perspective: The paper calls for a shift in research focus towards developing methods that integrate both interpretability and explainability, rather than treating them as mutually exclusive.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Interpretability and Explainability are not interchangeable: While often conflated, they represent distinct concepts. Interpretability is about understanding a model in itself, while explainability is about generating external descriptions of a model's behavior.
  • The "black-box" problem is real, but so are the limitations of "white-box" models: Relying solely on explainability for complex models means trusting potentially flawed or simplified explanations. Conversely, relying solely on interpretable models might mean sacrificing performance or missing crucial nuances that explainability could uncover.
  • Complementarity is key: An interpretable model can act as a sanity check for explanations, reducing the need for blind trust. Conversely, explainability techniques can help identify which features are most important for an interpretable model, aiding in its development and refinement.
  • Subjectivity of Interpretability: Interpretability is inherently human-centric and thus subjective. Explainability, by providing concrete (though potentially imperfect) outputs, can help ground this subjectivity and lead to more objective assessments.
  • Training Challenges: While enforcing interpretability can make model training more difficult, explainability tools can assist in this process, for example, by guiding feature selection or parameter pruning.

Key Results

This paper is conceptual and does not present empirical results in the traditional sense. Its "results" are in the form of a reasoned argument and a proposed shift in perspective. The key findings are the identification of specific flaws in each approach and the articulation of how their combination can address these flaws. For example:

  • Explainability Flaws Mitigated:
    • Blind trust in explanations (Problem E.1) is reduced because an interpretable model can be used to verify them.
  • Misalignment between explainer and explainee (Problem E.2) is lessened by having an objective, interpretable model as a common ground.
  • Explanations being "wrong" (Problem E.3) becomes less of an issue as they can be more faithful to a simpler, interpretable model.
  • Explanations raising more questions than answers (Problem E.4) can be resolved by consulting the interpretable model.
  • Interpretability Flaws Mitigated:
    • Incomplete information from interpretability alone (Problem I.1) can be supplemented by explainability techniques that provide feature importance, visualizations, etc.
  • Subjectivity of interpretability (Problem I.2) is reduced as explainability can offer more objective metrics and insights.
  • Training difficulties for interpretable models (Problem I.3) can be eased by using explainability tools for tasks like feature selection.
  • Interpretability alone not being enough for common ends (Problem I.4) is addressed by explainability's ability to compute metrics for fairness, bias detection, etc.

Limitations and Future Work

The authors acknowledge that this is a position paper and call for empirical verification of their claims. They suggest future work should focus on:

  • Empirically verifying the benefits of combining interpretability and explainability across various topics, including robustness, algorithmic stability, fairness, and data privacy.
  • Developing new methods that simultaneously leverage the assets of both interpretability and explainability in practical applications.
  • Further exploring the nuances of how these concepts interact and how they can be best integrated to build more trustworthy and effective AI systems.

Summary updated

Abstract

Interpretability and explainability have gained more and more attention in the field of machine learning as they are crucial when it comes to high-stakes decisions and troubleshooting. Since both provide information about predictors and their decision process, they are often seen as two independent means for one single end. This view has led to a dichotomous literature: explainability techniques designed for complex black-box models, or interpretable approaches ignoring the many explainability tools. In this position paper, we challenge the common idea that interpretability and explainability are substitutes for one another by listing their principal shortcomings and discussing how both of them mitigate the drawbacks of the other. In doing so, we call for a new perspective on interpretability and explainability, and works targeting both topics simultaneously, leveraging each of their respective assets.
BibTeX
@misc{leblanc2023relationship,
  author = {Benjamin Leblanc},
  title = {{On the Relationship Between Interpretability and Explainability in Machine Learning}},
  year = {2023},
  eprint = {2311.11491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11491},
}
Figure