Towards Robust Interpretability with Self-Explaining Neural Networks

David Alvarez-Melis, Tommi S. Jaakkola

TL;DR

Most AI "explanations" are generated after a model is trained, making them fragile and often inconsistent. This paper introduces Self-Explaining Neural Networks (SENN), a class of models that are interpretable by design. By enforcing "stability" through a novel regularization scheme, SENN ensures that similar inputs receive similar, human-understandable explanations without sacrificing predictive power.

The Problem: Post-hoc Instability

In the quest to understand "black box" deep learning, most researchers use post-hoc methods like LIME or SHAP. These tools take a pre-trained model and try to guess why it made a specific decision.

The authors identify a major flaw in this approach: instability. Because these explanations are calculated after the fact, they aren't tethered to the model's actual internal logic in a robust way. A tiny, imperceptible change to an input (like adding a bit of Gaussian noise) might not change the model's prediction, but it can lead to a wildly different explanation. This lack of robustness is a dealbreaker for high-stakes domains like medicine or law, where practitioners need to trust that the "why" behind a decision is consistent.

Methodology: Self-Explaining Architecture

The authors propose moving away from post-hoc explanations toward self-explaining models. They build SENN by generalizing the classic linear regression model ($f(x) = w^T x$) into a more flexible but still structured form:

$$f(x) = \sum_{i=1}^{k} \theta_i(x) h_i(x)$$

Architecture of the Self-Explaining Neural Network (SENN), featuring a concept encoder, relevance parametrizer, and aggregator.

The architecture consists of three main components:

  1. Concept Encoder ($h$): Maps the input $x$ to a small set of "basis concepts." For images, these might be high-level patterns like "diagonal strokes" or "circular shapes."
  2. Relevance Parametrizer ($\theta$): Acts like the "weights" in a linear model, but they are input-dependent. It tells us how much each concept contributes to the final prediction for this specific input.
  3. Aggregator ($g$): A simple, usually additive function that combines the concepts and weights to produce a prediction.

Robustness Regularization To ensure the model is truly interpretable, the authors enforce three "desiderata": explicitness, faithfulness, and stability. They achieve stability by adding a gradient-based regularization term to the loss function. This forces the weights ($\theta$) to vary slowly relative to the concepts ($h$), effectively making the model behave like a simple linear model in any local neighborhood.

Datasets and Evaluation

The authors evaluated SENN across a variety of tasks to prove its versatility:

  • Image Classification: MNIST (handwritten digits) and CIFAR-10 (objects).
  • Tabular/UCI Benchmarks: Ionosphere, Heart, Diabetes, Abalone, Glass, and Wine datasets.
  • Fairness/Risk Assessment: ProPublica’s COMPAS dataset, a standard benchmark for studying algorithmic bias.

Key Insights

  • Stability is a Metric of Trust: The authors argue that if an explanation isn't stable, it isn't useful. They formalize this using a "Lipschitz constant" for explanations—a mathematical way to measure how much an explanation changes when the input moves slightly.
  • The Accuracy-Interpretability Trade-off is a False Dichotomy: SENN achieves near-state-of-the-art accuracy (e.g., >99% on MNIST) while remaining fully transparent.
  • Concepts > Raw Features: Explaining an image by highlighting individual pixels is often just "visual noise." By learning a "basis" of concepts and grounding them with prototypes (real examples from the training set), the explanations become much more intuitive for humans.

Key Results

  • Superior Robustness: In "adversarial" tests for interpretability, SENN's explanations remained consistent under perturbations that caused post-hoc methods like LIME and SHAP to produce nonsensical results.
  • Quantitative Stability: SENN consistently achieves lower Lipschitz estimates compared to post-hoc methods, indicating significantly higher explanation stability across various datasets.
Lipschitz estimates for SENN vs. post-hoc methods across UCI datasets, demonstrating superior explanation stability.
  • Competitive Performance: On the COMPAS dataset, SENN outperformed a baseline logistic regression model (82.02% vs 78.54% accuracy) while providing deeper insight into risk drivers.
  • Faithfulness: Through "feature removal" tests, the authors showed that the features SENN identified as "important" were indeed the ones the model relied on most for its predictions.

Limitations and Future Work

  • Concept Grounding: While the model learns concepts automatically, "grounding" them still relies on looking at prototypes, which can be subjective.
  • Complexity of Encoders: For very complex datasets, the concept encoder itself might become a complex sub-network, potentially shifting the "black box" problem one level deeper.
  • Scaling: Future work involves testing these models on much larger, real-world image datasets and natural language processing (NLP) tasks.

Summary updated

BibTeX
@inproceedings{alvarezmelis2018towards,
  author = {David Alvarez-Melis and Tommi S. Jaakkola},
  title = {{Towards Robust Interpretability with Self-Explaining Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=H1bm8vb_ZB},
}
Figure