Interpretability in Machine Learning: on the Interplay with Explainability, Predictive Performances and Models

Benjamin Leblanc, Pascal Germain

Introduction

In high-stakes domains such as medicine and finance, the deployment of machine learning models is often hindered by their "black-box" nature. This lack of transparency makes auditing for biases or errors difficult, leading to a divide in the research community. On one side, the Explainability School advocates for complex, high-performing models paired with post-hoc explanation tools like SHAP or LIME. On the other, the Interpretability School promotes inherently simple models—such as small decision trees or linear regressions—that humans can inspect directly. This paper argues that this "either-or" dichotomy is counterproductive and proposes a synergistic framework: the Explained Interpretable Predictor.

The Limitations of Siloed Approaches

Current methodologies often treat interpretability and explainability as substitutes, yet both possess inherent flaws when used in isolation.

The "Blind Trust" in Explainability

Post-hoc explainability tools are often "necessarily wrong" simplifications. A significant risk is the "obligation to blindly trust" these tools without a ground truth. For instance, saliency map techniques can produce visually convincing explanations even for models that have learned nothing meaningful.

Saliency maps for models trained on true vs. random labels, demonstrating the risk of blind trust in explainability tools.

As shown in the MNIST example above, different saliency methods can yield similar-looking explanations for models trained on true labels versus those trained on completely random labels. This highlights that explanations can be decoupled from the actual model logic, making them potentially misleading.

The Subjectivity of Interpretability

Conversely, "interpretable" models are not a panacea. What one human finds "simple" is often subjective. Furthermore, simple models may fail to capture the nuances of the underlying data.

A linear regression line fitted to a non-linear dataset from Anscombe’s Quartet, showing that model simplicity does not guarantee a complete understanding of data relationships.

Using a simple linear model (as seen in the Anscombe’s Quartet example above) might provide a clear mathematical formula, but it fails to reveal the true non-linear relationship in the data. Relying solely on the model's "transparency" can thus lead to a false sense of security.

A Synergistic Framework: Explained Interpretable Predictors

The paper proposes bridging the gap by using both approaches simultaneously. In this framework, interpretability provides a verifiable ground truth to audit explanations, while explainability tools offer deep diagnostic insights that simple models might otherwise lack.

  • Interpretability as a Sanity Check: If the underlying model is transparent, a human can verify if a post-hoc explanation (like feature importance) aligns with the model's actual structure.
  • The "Common Ground" Effect: An interpretable model acts as an objective reference point for both the explainer and the user, reducing communication misalignment.
  • Explainability as a Training Tool: Post-hoc tools can be used during the training phase of interpretable models—for example, using SHAP values to guide feature pruning and maintain model parsimony.

Conclusion and Future Work

The shift toward Explained Interpretable Predictors offers several advantages, including reduced subjectivity through objective statistics, increased robustness against adversarial attacks, and lower computational costs for generating explanations.

However, this theoretical framework requires empirical validation. Future research should focus on how this synergy impacts algorithmic fairness, model stability over time, and, most importantly, human performance in real-world decision-making tasks. By moving beyond the "either-or" debate, the ML community can develop systems that are both high-performing and truly trustworthy.

Summary updated

BibTeX
@misc{leblanc2023interpretability,
  author = {Benjamin Leblanc and Pascal Germain},
  title = {{Interpretability in Machine Learning: on the Interplay with Explainability, Predictive Performances and Models}},
  year = {2023},
  eprint = {2311.11491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11491},
}
Figure