From Heatmaps to Structural Explanations of Image Classifiers

Fuxin Li, Zhongang Qi, Saeed Khorram, Vivswan Shitole, Prasad Tadepalli, Minsuk Kahng, Alan Fern

Abstract

Explainable AI (XAI) has traditionally relied on saliency maps to interpret deep neural networks. However, these heatmaps often fail to capture the complexity of model reasoning and are susceptible to adversarial noise. This paper summarizes research into more robust frameworks, including Explainable Neural Networks (XNN) and Structured Attention Graphs (SAGs), which provide multi-faceted, manifold-aligned explanations for image classifiers.

Introduction

Deep neural networks are notoriously opaque "black boxes." While saliency maps have been the standard for visualization, they suffer from human projection bias and the "single-explanation trap." Research from Oregon State University suggests that models often rely on multiple independent features for a single classification, a nuance that standard heatmaps fail to capture. This research moves toward structured explanations that better reflect the internal logic of the model while ensuring explanations stay on the "data manifold."

Methodological Frameworks

The research team developed three key methodologies to improve interpretability:

Explainable Neural Networks (XNN)

XNNs use Sparse Reconstruction Autoencoders (SRAE) to identify "x-features"—high-level concepts extracted from intermediate layers. This allows the model to explain itself using its own internal representations. For example, in bird classification, different x-features might correspond to the head or the wing patterns independently.

XNN visualizations showing distinct x-features for Downy Woodpeckers.

iGOS++ and the Data Manifold

To prevent "adversarial explanations"—where optimization finds pixel perturbations that change confidence without identifying semantic features—iGOS++ utilizes Integrated Gradients and Bilateral Total-Variance (BTV) regularization. This ensures that generated masks remain natural and aligned with the data manifold, preventing the model from being misled by out-of-distribution noise.

Structured Attention Graphs (SAGs)

SAGs represent the latest evolution in this research. Utilizing a beam search algorithm, SAGs identify multiple Minimal Sufficient Explanations (MSEs). These are visualized as a directed graph, illustrating how different combinations of image patches independently or jointly contribute to a classification decision.

Key Insights and Empirical Results

Research across datasets like ImageNet and CUB-200 yielded several critical insights:

  • Multiplicity of Explanations: For 30% of ImageNet images, the model relies on more than one independent set of features for classification, proving that a single heatmap is fundamentally insufficient.
  • Locality Assumption: In 80% of cases, only 20% of the image area is necessary for the model to reach its decision.
  • Human Performance: XNN visualizations improved human accuracy in fine-grained bird classification from 56% to 72%. SAGs improved the accuracy of users' mental models of the AI from 57% to 86%.

Debugging via XAI

XAI tools are essential for identifying "Clever Hans" effects, where models learn the wrong cues. In a study of COVID-19 X-ray classifiers, iGOS++ revealed that a high-performing model was actually achieving success by identifying printed text labels on the X-ray film rather than pathological features in the lungs.

Grad-CAM heatmap revealing a model focusing on text labels rather than clinical features in COVID-19 X-rays.

Conclusion and Future Work

Transitioning from heatmaps to structured explanations significantly enhances the utility of XAI and the reliability of human-AI collaboration. While these methods increase cognitive load, they provide a more faithful representation of model behavior. Future research will explore formal causality and applications in reinforcement learning and 3D point clouds.

Summary updated

BibTeX
@article{li2021from,
  author = {Fuxin Li and Zhongang Qi and Saeed Khorram and Vivswan Shitole and Prasad Tadepalli and Minsuk Kahng and Alan Fern},
  title = {{From Heatmaps to Structural Explanations of Image Classifiers}},
  journal = {Applied AI Letters},
  year = {2021},
  doi = {10.1002/ail2.46},
}
Figure