A Benchmark for Interpretability Methods in Deep Neural Networks

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, Been Kim

TL;DR

The paper introduces ROAR (RemOve And Retrain), a rigorous benchmark for evaluating interpretability methods in deep neural networks. By retraining models from scratch after removing "important" features, the authors reveal a startling truth: many popular saliency methods are no more informative than a random guess, while specific ensemble approaches like VarGrad and SmoothGrad-Squared actually provide meaningful insights.

What Problem Does This Paper Address?

Deep neural networks are notoriously "black boxes." To peek inside, researchers use interpretability methods (like saliency maps) to identify which input features (pixels) are most important for a prediction. However, evaluating these methods is notoriously difficult because there is no "ground truth" for what a model actually finds important.

The standard evaluation approach has been to remove pixels deemed "important" and observe the drop in a pre-trained model's accuracy. The authors argue this is fundamentally flawed due to distribution shift. If you mask pixels with a constant value (like black or the mean), the model is suddenly seeing data it never encountered during training. The resulting drop in accuracy might be because the model is "confused" by the artifacts of the mask, rather than because it actually lost vital information.

Methods: The ROAR Benchmark

The authors propose ROAR (RemOve And Retrain). The process is straightforward but computationally intensive:

  1. Rank: Use an interpretability method (e.g., Integrated Gradients) to rank pixels by importance for every image in the dataset.
  2. Modify: Replace the top $t\%$ of pixels with the dataset's per-channel mean.
  3. Retrain: Train a new model from scratch on this modified dataset.
  4. Measure: Evaluate the accuracy of this new model.
Visualizing feature removal

If an interpretability method is accurate, removing the pixels it identifies as "important" should cause a sharper decline in the retrained model's accuracy compared to removing pixels randomly. By retraining, the authors ensure the model "learns" to deal with the modified distribution, decoupling the loss of information from the shock of seeing masked pixels.

The paper evaluates several Base Estimators (GRAD, Integrated Gradients, Guided Backprop) and Ensemble Estimators (SmoothGrad, SmoothGrad-Squared, VarGrad).

Datasets

The benchmark was tested across three large-scale image classification datasets:

  • ImageNet: The gold standard for computer vision.
  • Food 101: A dataset of 101 food categories.
  • Birdsnap: A fine-grained dataset of North American bird species.

The authors also used a 16-dimensional synthetic dataset to validate ROAR in a controlled environment where the "ground truth" importance was known.

Key Insights and Ideas

  • CNNs are incredibly robust: One of the most surprising findings was that you can remove 90% of pixels from ImageNet images and a retrained ResNet-50 can still achieve 63.53% accuracy (down from 76.68%). This suggests that images contain massive amounts of redundant information.
  • Base methods fail the test: Popular methods like Integrated Gradients (IG) and Guided Backprop (GB) often performed worse than a random baseline when evaluated via ROAR. This suggests that the "explanations" these methods provide may not actually reflect the features the model relies on for generalization.
  • Ensembling is the "Secret Sauce": While base methods struggled, ensembling techniques that aggregate multiple noisy estimates—specifically VarGrad and SmoothGrad-Squared—significantly outperformed random baselines.
Impact of retraining on accuracy
  • The "Retrain" step is non-negotiable: Without retraining, even a random mask causes a massive accuracy drop. ROAR proves that much of the "success" previously attributed to interpretability methods was actually just the model reacting to out-of-distribution artifacts.

Key Results

  • Random vs. Base: On ImageNet, at 90% feature removal, the random baseline resulted in a model with ~63.5% accuracy. Base estimators like GRAD and IG resulted in models with lower accuracy (~60-62%), meaning they were slightly better than random at finding "useful" pixels to remove, but the gap was surprisingly narrow.
  • The Winners: VarGrad and SmoothGrad-Squared (SG-SQ) showed the steepest degradation curves. On ImageNet at 90% removal, SG-SQ reduced accuracy to ~29%, proving it was far more effective at identifying the "core" features the model needs to function.
  • SmoothGrad Paradox: Interestingly, "Classic" SmoothGrad (which averages gradients) often performed worse than a single estimate, while squaring the gradients before averaging (SG-SQ) or taking the variance (VarGrad) was highly effective.

Limitations and Future Work

  • Computational Cost: The primary limitation is that ROAR requires retraining a model from scratch for every percentage threshold and every method. This is extremely expensive for large datasets like ImageNet.
  • Feature Redundancy: ROAR may be less sensitive if a dataset has high redundancy (e.g., the same information is present in multiple pixels). The authors suggest that in such cases, accuracy might not drop until all redundant features are removed.
  • Future Directions: The authors suggest investigating why squaring gradients or taking the variance (as in VarGrad) works so much better than simple averaging. They also open the door for ROAR to be applied to other domains like NLP or medical imaging where feature importance is critical.

Summary updated

Abstract

We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.
BibTeX
@inproceedings{hooker2019benchmark,
  author = {Sara Hooker and Dumitru Erhan and Pieter-Jan Kindermans and Been Kim},
  title = {{A Benchmark for Interpretability Methods in Deep Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=zQwW-VHVEc},
}
Figure