Explainable Models with Consistent Interpretations

Vipin Pillai, Hamed PirsiavashAAAI 2021 · OpenReview

Abstract

Given the widespread deployment of black box deep neural networks in computer vision applications, the interpretability aspect of these black box systems has recently gained traction. Various methods have been proposed to explain the results of such deep neural networks. However, some recent works have shown that such explanation methods are biased and do not produce consistent interpretations. Hence, rather than introducing a novel explanation method, we learn models that are encouraged to be interpretable given an explanation method. We use Grad-CAM as the explanation algorithm and encourage the network to learn consistent interpretations along with maximizing the log-likelihood of the correct class. We show that our method outperforms the baseline on the pointing game evaluation on ImageNet and MS-COCO datasets respectively. We also introduce new evaluation metrics that penalize the saliency map if it lies outside the ground truth bounding box or segmentation mask, and show that our method outperforms the baseline on these metrics as well. Moreover, our model trained with interpretation consistency generalizes to other explanation algorithms on all the evaluation metrics. The code and models are publicly available.

Introduction: The Black Box Problem

Deep neural networks (DNNs) are notoriously difficult to interpret. While methods like Grad-CAM produce saliency maps to explain decisions, these explanations often reveal that models rely on association bias—identifying a "fish" by the human hand holding it rather than the fish itself. This paper argues that the issue lies not just in the explanation methods, but in the models themselves. By enforcing Interpretation Consistency during training, we can create models that are "right for the right reasons."

Methodology: Enforcing Consistency

The authors propose a training constraint called Enforcing Consistency in Interpretation (GC). The pipeline involves three key components:

  1. Image Gridding: A $2 \times 2$ composite image is created where only one quadrant contains the target class (e.g., a keyboard).
  2. Consistency Loss ($L_{GC}$): The model is penalized if any target-class saliency appears in the three distractor quadrants. This forces the model to localize its reasoning within the correct spatial bounds.
  3. Global Max Pooling (GMP): Replacing standard Global Average Pooling (GAP) with GMP encourages the model to focus on the most discriminative, concise regions of an object rather than spreading attention across the entire image.
The proposed training pipeline using image gridding and consistency loss to enforce spatial interpretability.

Results and Insights

The approach was evaluated on ImageNet and MS-COCO using metrics like the Pointing Game (PG) and a new Stochastic Pointing Game (SPG), which measures the cleanliness of the saliency distribution.

  • Sharper Localization: Qualitative results show that the combination of GC and GMP leads to significantly more focused heatmaps that stay within object boundaries. For instance, on ImageNet, AlexNet's SPG score jumped from 53.45% to 61.15%.
  • Adversarial Robustness: By forcing the model to ignore spurious background features, it becomes more robust to adversarial attacks. In a ResNet50, the GMP + GC model maintained higher accuracy (2.71%) compared to the baseline (1.06%) under extreme adversarial noise.
  • Generalization: Improvements carry over to other explanation methods like FullGrad, proving the model fundamentally changes how it learns features rather than just "gaming" Grad-CAM.
Qualitative comparison of saliency maps across different models. The GMP + GC approach produces the most concise and accurate localizations.

Conclusion

By treating interpretability as a first-class training constraint, this work moves beyond merely explaining black boxes to building models that are inherently more trustworthy. While the "double gradient" calculation adds computational overhead, the gains in localization accuracy and adversarial robustness provide a strong argument for enforcing interpretation consistency in safety-critical AI applications.

Summary updated

Abstract

Given the widespread deployment of black box deep neural networks in computer vision applications, the interpretability aspect of these black box systems has recently gained traction. Various methods have been proposed to explain the results of such deep neural networks. However, some recent works have shown that such explanation methods are biased and do not produce consistent interpretations. Hence, rather than introducing a novel explanation method, we learn models that are encouraged to be interpretable given an explanation method. We use Grad-CAM as the explanation algorithm and encourage the network to learn consistent interpretations along with maximizing the log-likelihood of the correct class. We show that our method outperforms the baseline on the pointing game evaluation on ImageNet and MS-COCO datasets respectively. We also introduce new evaluation metrics that penalize the saliency map if it lies outside the ground truth bounding box or segmentation mask, and show that our method outperforms the baseline on these metrics as well. Moreover, our model trained with interpretation consistency generalizes to other explanation algorithms on all the evaluation metrics. The code and models are publicly available.
BibTeX
@inproceedings{pillai2021explainable,
  author = {Vipin Pillai and Hamed Pirsiavash},
  title = {{Explainable Models with Consistent Interpretations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=2lmiSBJlHca},
}
Figure