Abstract
Explainable AI (XAI) has traditionally relied on saliency maps to interpret deep neural networks. However, these heatmaps often fail to capture the complexity of model reasoning and are susceptible to adversarial noise. This paper summarizes research into more robust frameworks, including Explainable Neural Networks (XNN) and Structured Attention Graphs (SAGs), which provide multi-faceted, manifold-aligned explanations for image classifiers.
Introduction
Deep neural networks are notoriously opaque "black boxes." While saliency maps have been the standard for visualization, they suffer from human projection bias and the "single-explanation trap." Research from Oregon State University suggests that models often rely on multiple independent features for a single classification, a nuance that standard heatmaps fail to capture. This research moves toward structured explanations that better reflect the internal logic of the model while ensuring explanations stay on the "data manifold."
Methodological Frameworks
The research team developed three key methodologies to improve interpretability:
Explainable Neural Networks (XNN)
XNNs use Sparse Reconstruction Autoencoders (SRAE) to identify "x-features"—high-level concepts extracted from intermediate layers. This allows the model to explain itself using its own internal representations. For example, in bird classification, different x-features might correspond to the head or the wing patterns independently.
iGOS++ and the Data Manifold
To prevent "adversarial explanations"—where optimization finds pixel perturbations that change confidence without identifying semantic features—iGOS++ utilizes Integrated Gradients and Bilateral Total-Variance (BTV) regularization. This ensures that generated masks remain natural and aligned with the data manifold, preventing the model from being misled by out-of-distribution noise.
Structured Attention Graphs (SAGs)
SAGs represent the latest evolution in this research. Utilizing a beam search algorithm, SAGs identify multiple Minimal Sufficient Explanations (MSEs). These are visualized as a directed graph, illustrating how different combinations of image patches independently or jointly contribute to a classification decision.
Key Insights and Empirical Results
Research across datasets like ImageNet and CUB-200 yielded several critical insights:
- Multiplicity of Explanations: For 30% of ImageNet images, the model relies on more than one independent set of features for classification, proving that a single heatmap is fundamentally insufficient.
- Locality Assumption: In 80% of cases, only 20% of the image area is necessary for the model to reach its decision.
- Human Performance: XNN visualizations improved human accuracy in fine-grained bird classification from 56% to 72%. SAGs improved the accuracy of users' mental models of the AI from 57% to 86%.
Debugging via XAI
XAI tools are essential for identifying "Clever Hans" effects, where models learn the wrong cues. In a study of COVID-19 X-ray classifiers, iGOS++ revealed that a high-performing model was actually achieving success by identifying printed text labels on the X-ray film rather than pathological features in the lungs.
Conclusion and Future Work
Transitioning from heatmaps to structured explanations significantly enhances the utility of XAI and the reliability of human-AI collaboration. While these methods increase cognitive load, they provide a more faithful representation of model behavior. Future research will explore formal causality and applications in reinforcement learning and 3D point clouds.
Summary updated