Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability

Roman Levin, Manli Shu, Eitan Borgnia, Furong Huang, Micah Goldblum, Tom Goldstein

Most explainability tools in computer vision, such as Grad-CAM or Integrated Gradients, produce input-space saliency maps. These tools highlight which pixels in an image most heavily influenced a model's prediction. However, knowing where a model looked doesn't always tell you why it failed. A model might look at the correct object but still misclassify it because a specific internal filter is poorly tuned, or because it is over-relying on a spurious correlation (like associating a boat in the background with a shark).

To address this gap, researchers are shifting focus from the input image to the network's internal parameters. Instead of asking "which pixels did the model look at?", this approach asks "which parts of the model's 'brain' are broken?" By analyzing the gradients of the loss with respect to network parameters, we can identify specific components responsible for misclassifications, allowing us to "surgically" fix errors and find semantically similar failure modes.

The Parameter-Space Approach

The authors introduce Parameter-Space Saliency Maps through a three-step diagnostic process:

  1. Filter-wise Saliency: The process begins by calculating the magnitude of the gradient of the loss with respect to every parameter for a given misclassified sample. To make this massive amount of data interpretable, the values are aggregated at the filter level (averaging the gradients of all weights within a single convolutional filter).
  2. Standardization: Raw gradients are naturally larger in shallower layers. To find what is uniquely wrong with a specific sample, the saliency profile is standardized against the entire validation set. This highlights filters that are "abnormally" active for a specific error, rather than just filters that are generally important for all images.
  3. Input-Space Counterpart: Once the "broken" filters are identified, an inversion technique maps those filters back to the input pixels. This reveals which specific image features triggered those specific filters to malfunction.
Average saliency profiles of correctly vs. incorrectly classified samples across layers.

Standardization is the "secret sauce" of this method. Without it, saliency maps almost always point to the first few layers of a network. As shown in the figure above, standardizing reveals that for misclassified samples, the "culprit" filters are often located much deeper in the network, exhibiting massive spikes in abnormal activity compared to correctly classified samples.

Key Insights: Saliency Neighbors and Surgical Fixes

By mapping errors to specific parameters, the researchers uncovered several fascinating behaviors within neural networks:

  • Saliency Neighbors are Semantically Linked: If two different images trigger the same salient filters to malfunction, they are considered "neighbors" in parameter space. These neighbors often share the same semantic reason for failure, such as a missing visual cue (like an animal's legs being hidden) or a confusing background element.
  • Spurious Correlation Detection: The input-space counterpart can pinpoint exactly which background objects are confusing the model. In one case study, a "Great White Shark" was misclassified as a "Killer Whale" not because of the shark itself, but because the model associated the combination of a "seal" and a "boat" in the background with whales.
  • Surgical Fine-Tuning: Updating just a tiny fraction of the network—the top 0.6% of salient filters—on a single image was found to be as effective at correcting errors as fine-tuning the entire network. Furthermore, fine-tuning these filters on a single misclassified image generalized to correct errors in its "saliency neighbors" on an independent test set (ImageNet-v2).

Key Results: Pruning for Profit

To prove that the identified filters were truly responsible for the errors, the authors conducted pruning experiments.

Effects of pruning salient filters on class confidence and error correction.

As demonstrated in the charts above, pruning (zeroing out) the top 100 most salient filters for a misclassified image resulted in a massive 30% drop in incorrect class confidence, compared to only a ~7% drop when pruning random filters. Even more impressively, simply "turning off" the 30 most salient filters corrected the labels for 12% of misclassified samples without any additional training.

Limitations and Future Directions

While highly effective as a diagnostic tool, this approach has limitations. The current work focuses heavily on Convolutional Neural Networks (CNNs) and filter-level aggregation. While the underlying theory should hold for other architectures, it remains to be seen how this translates to Vision Transformers (ViTs), where attention heads or MLP blocks would be the focus instead of filters. Additionally, calculating gradients for every parameter across a validation set to create the standardization baseline can be computationally expensive for extremely large models.

Ultimately, parameter-space saliency maps offer a powerful new lens for model debugging. Future work could leverage these maps not just for post-hoc analysis, but to automatically identify and regularize "biased" or malfunctioning filters during the training process itself.

Summary updated

Abstract

Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We first verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples more than turning off the same number of random or least salient parameters. We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassified for similar reasons -- nearest neighbors in the saliency space. After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar. Further, we introduce an input-space saliency counterpart which reveals how image features cause specific network components to malfunction.
BibTeX
@inproceedings{levin2022where,
  author = {Roman Levin and Manli Shu and Eitan Borgnia and Furong Huang and Micah Goldblum and Tom Goldstein},
  title = {{Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=FbtnVLnjXtI},
}
Figure