The Intriguing Properties of Model Explanations

Maruan Al-Shedivat, Avinava Dubey, Eric P. XingarXiv 2018 · OpenReview

Abstract

Linear approximations to the decision boundary of a complex model have become one of the most popular tools for interpreting predictions. In this paper, we study such linear explanations produced either post-hoc by a few recent methods or generated along with predictions with contextual explanation networks (CENs). We focus on two questions: (i) whether linear explanations are always consistent or can be misleading, and (ii) when integrated into the prediction process, whether and how explanations affect the performance of the model. Our analysis sheds more light on certain properties of explanations produced by different methods and suggests that learning models that explain and predict jointly is often advantageous.

Introduction

As deep learning models grow in complexity, the demand for interpretability has led to the widespread adoption of post-hoc explanation tools like LIME. These tools approximate a "black-box" model's behavior using simple linear models. However, the reliability of these approximations and their impact on model performance remain poorly understood. This research investigates whether linear explanations can be misleading and how integrating explanations into the training process affects predictive accuracy.

Methods: Post-hoc vs. Joint Learning

The study compares two primary paradigms for model interpretability:

  1. Post-hoc Explanations (e.g., LIME): These methods operate after a model is trained, searching for a local linear approximation of the decision boundary. While model-agnostic, they are decoupled from the actual prediction logic.
  2. Contextual Explanation Networks (CENs): CENs integrate explanations directly into the architecture. A deep neural network (the encoder) processes input $x$ to generate weights for a linear model, which then uses interpretable attributes $z$ to produce the final prediction. Here, the prediction is the explanation.

The Overfitting Trap of Post-hoc Explanations

A critical finding of this research is that post-hoc methods like LIME can "overfit" to a model's decision boundary. When provided with low-quality or even purely noisy features, LIME can still produce a seemingly consistent linear approximation. This creates a false sense of security, as the tool might provide a confident-looking explanation based on irrelevant data.

In contrast, CENs act as an "honesty check." Since they rely on interpretable features for the actual prediction, their performance drops significantly when those features are noisy. This makes CENs a more reliable diagnostic tool for assessing feature quality.

Figure 5: Test error vs. Signal-to-Noise Ratio (SNR) on IMDB. While LIME's error remains relatively stable (hiding the noise), CEN error increases as the signal quality degrades, reflecting the true utility of the features.

Explanations as a Regularizer

Forcing a model to explain itself serves as a powerful constraint. The authors found that CENs often outperform standard black-box models like CNNs or LSTMs, particularly when training data is scarce. By requiring the model to map its internal representations to a simple linear model of interpretable attributes, the "explanation layer" prevents the model from overfitting to noise in small datasets.

Figure 1: (a) Training error over time and (b) Validation error vs. training set size. CENs demonstrate superior sample efficiency, maintaining lower error rates than standard CNNs on MNIST when trained on as little as 1% of the data.

Key Results and Real-World Application

  • Sample Efficiency: On MNIST, CENs trained on only 1% of the data maintained significantly lower validation error than standard CNNs.
  • Predictive Accuracy: On the IMDB dataset, a CEN using bag-of-words features achieved a 6.9% error rate, nearly halving the 13.2% error of a standard LSTM.
  • Context-Dependent Logic: In a poverty prediction task using satellite imagery, CENs automatically discovered distinct explanation models for urban vs. rural areas, focusing on nightlight intensity and agricultural features respectively.

Conclusion

While post-hoc explanations are popular, they risk providing misleadingly consistent results on poor data. Contextual Explanation Networks offer a more robust alternative, providing reliable diagnostics and acting as a regularizer that improves performance and sample efficiency. Future work may extend this joint learning framework to more complex explanation families beyond linear models.

Summary updated

Abstract

Linear approximations to the decision boundary of a complex model have become one of the most popular tools for interpreting predictions. In this paper, we study such linear explanations produced either post-hoc by a few recent methods or generated along with predictions with contextual explanation networks (CENs). We focus on two questions: (i) whether linear explanations are always consistent or can be misleading, and (ii) when integrated into the prediction process, whether and how explanations affect the performance of the model. Our analysis sheds more light on certain properties of explanations produced by different methods and suggests that learning models that explain and predict jointly is often advantageous.
BibTeX
@misc{alshedivat2018intriguing,
  author = {Maruan Al-Shedivat and Avinava Dubey and Eric P. Xing},
  title = {{The Intriguing Properties of Model Explanations}},
  year = {2018},
  eprint = {1801.09808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.09808},
}
Figure