Abstract
While saliency maps identify regions of interest in Convolutional Neural Network (CNN) predictions, they often fail to explain why a model misclassifies an object it appears to be focusing on. This paper introduces a framework based on "Surprise" and "Expectation" to diagnose failures by comparing real-time filter activations against training distributions. By identifying misleading "surprising" features or missing "expected" features, the authors provide a deeper diagnostic tool for black-box models and demonstrate a method for correcting misclassifications.
The Saliency Gap
CNNs are often criticized as "black boxes." Existing interpretability techniques like Grad-CAM, RISE, or LIME produce heatmaps showing where a network is looking. However, these methods fall short when a model fails. For instance, if a network misclassifies a goldfish as a "roundworm," a standard saliency map might still highlight the fish perfectly. This leaves researchers asking: "If the model is looking at the right thing, why did it get the wrong answer?"
The gap lies in understanding the relationship between a specific input and the model's internalized knowledge of a class gained during training. Traditional maps show attention, but not the reasoning behind the final label.
Methodology: Surprise and Expectation
The authors propose a framework based on the statistical distribution of filter activations. By recording how every filter in the final convolutional layer behaves for every class in the training set, they create a "normal" distribution of activations.
- Grad-AMap: A new filter importance measure calculating the mean product of a filter’s activation and its gradient to identify influential filters.
- Surprise: Occurs when a filter activates much more strongly than it did during training for that class (above the 99.9th percentile).
- Expectation: Occurs when a filter activates much more weakly than expected (below the 0.1st percentile).
- The $\beta$ Value: A global metric measuring how "typical" an image is. A high $\beta$ aligns with training data; a low $\beta$ suggests the image is atypical.
Key Insights and Results
The "Why" of Failure
The authors found that failures generally fall into two categories. High $\beta$ failures occur when an image is very similar to another class (e.g., a "Bighorn" sheep mistaken for a "Ram"). Low $\beta$ failures happen when the image is atypical—perhaps due to strange scales or unusual backgrounds. For example, a model might expect to see light-colored ground when identifying a "Gila Monster"; if the background is dark red, the "expectation" is not met, leading to failure.
Quantifying Performance
- Superior Ranking: The Grad-AMap technique outperformed existing methods like Grad-CAM and Taylor expansions in filter ranking, achieving the lowest Area Under the Curve (AUC) scores (0.042 for VGG16) in deletion tests.
- Failure Analysis: For VGG16, 86.6% of misclassified images showed a negative deviation from the mean ($\beta$), suggesting most failures occur because the image lacks features the network expects to see.
- Model Correction: By identifying "surprising" regions (distractions) and suppressing them, the authors "fixed" 5,116 misclassified images in the ImageNet validation set, outperforming standard backpropagation-based correction methods.
Conclusion
By moving beyond simple saliency to the concepts of Surprise and Expectation, this research provides a more nuanced understanding of CNN failure modes. While "fixing" missing expectations remains a challenge requiring generative approaches, the ability to identify and suppress distracting features offers a promising path toward more robust and interpretable computer vision systems.
Summary updated