Explaining Failure: Investigation of Surprise and Expectation in CNNs

Thomas Hartley, Kirill A. Sidorov, Christopher Willis, A. David MarshallWorkshops @ CVPR 2020 · DOI

Abstract

While saliency maps identify regions of interest in Convolutional Neural Network (CNN) predictions, they often fail to explain why a model misclassifies an object it appears to be focusing on. This paper introduces a framework based on "Surprise" and "Expectation" to diagnose failures by comparing real-time filter activations against training distributions. By identifying misleading "surprising" features or missing "expected" features, the authors provide a deeper diagnostic tool for black-box models and demonstrate a method for correcting misclassifications.

The Saliency Gap

CNNs are often criticized as "black boxes." Existing interpretability techniques like Grad-CAM, RISE, or LIME produce heatmaps showing where a network is looking. However, these methods fall short when a model fails. For instance, if a network misclassifies a goldfish as a "roundworm," a standard saliency map might still highlight the fish perfectly. This leaves researchers asking: "If the model is looking at the right thing, why did it get the wrong answer?"

Comparison of traditional saliency methods (Grad-CAM, RISE, LIME, etc.) on a fish classification task.

The gap lies in understanding the relationship between a specific input and the model's internalized knowledge of a class gained during training. Traditional maps show attention, but not the reasoning behind the final label.

Methodology: Surprise and Expectation

The authors propose a framework based on the statistical distribution of filter activations. By recording how every filter in the final convolutional layer behaves for every class in the training set, they create a "normal" distribution of activations.

  1. Grad-AMap: A new filter importance measure calculating the mean product of a filter’s activation and its gradient to identify influential filters.
  2. Surprise: Occurs when a filter activates much more strongly than it did during training for that class (above the 99.9th percentile).
  3. Expectation: Occurs when a filter activates much more weakly than expected (below the 0.1st percentile).
  4. The $\beta$ Value: A global metric measuring how "typical" an image is. A high $\beta$ aligns with training data; a low $\beta$ suggests the image is atypical.
Histogram showing the distribution of activation scores used to define thresholds for surprise and expectation.

Key Insights and Results

The "Why" of Failure

The authors found that failures generally fall into two categories. High $\beta$ failures occur when an image is very similar to another class (e.g., a "Bighorn" sheep mistaken for a "Ram"). Low $\beta$ failures happen when the image is atypical—perhaps due to strange scales or unusual backgrounds. For example, a model might expect to see light-colored ground when identifying a "Gila Monster"; if the background is dark red, the "expectation" is not met, leading to failure.

Quantifying Performance

  • Superior Ranking: The Grad-AMap technique outperformed existing methods like Grad-CAM and Taylor expansions in filter ranking, achieving the lowest Area Under the Curve (AUC) scores (0.042 for VGG16) in deletion tests.
  • Failure Analysis: For VGG16, 86.6% of misclassified images showed a negative deviation from the mean ($\beta$), suggesting most failures occur because the image lacks features the network expects to see.
  • Model Correction: By identifying "surprising" regions (distractions) and suppressing them, the authors "fixed" 5,116 misclassified images in the ImageNet validation set, outperforming standard backpropagation-based correction methods.

Conclusion

By moving beyond simple saliency to the concepts of Surprise and Expectation, this research provides a more nuanced understanding of CNN failure modes. While "fixing" missing expectations remains a challenge requiring generative approaches, the ability to identify and suppress distracting features offers a promising path toward more robust and interpretable computer vision systems.

Summary updated

BibTeX
@inproceedings{hartley2020explaining,
  author = {Thomas Hartley and Kirill A. Sidorov and Christopher Willis and A. David Marshall},
  title = {{Explaining Failure: Investigation of Surprise and Expectation in CNNs}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00014},
}
Figure