Abstract
Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We find that samples which cause similar parameters to malfunction are semantically similar. We also show that pruning the most salient parameters for a wrongly classified sample often improves model behavior. Furthermore, fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples that are misclassified for similar reasons. Based on our parameter saliency method, we also introduce an input-space saliency technique that reveals how image features cause specific network components to malfunction. Further, we rigorously validate the meaningfulness of our saliency maps on both the dataset and case-study levels.
Introduction
Explainability in deep learning has traditionally focused on input-space saliency. Tools like Grad-CAM or Integrated Gradients highlight pixels that influence a model's prediction. However, these maps often fail to identify the specific internal components responsible for errors. For instance, if a model misidentifies an object, an input map might highlight the object itself without revealing why the internal logic failed. This research shifts the focus from the input to the parameters, aiming to pinpoint the exact "broken" filters that trigger misclassifications.
Methodology: Parameter Saliency Profiles
The core contribution is the Parameter Saliency Profile. The process involves:
- Gradient Magnitude Calculation: For a misclassified image, the gradient of the loss function is calculated with respect to every parameter. High gradients indicate parameters to which the loss is highly sensitive—the "malfunctioning" components.
- Filter-wise Aggregation: Magnitudes are averaged across entire convolutional filters to improve interpretability.
- Standardization: To isolate filters uniquely responsible for an error, scores are standardized (z-scores) against a validation set. This distinguishes "naturally" active filters (like early edge detectors) from those specifically failing on a given input.
Key Insights: Semantic Failure Modes
A major finding is that failure modes are semantically consistent. Images sharing similar parameter saliency profiles are often misclassified for the same underlying reason. For example, if a model fails to recognize a specific breed of dog because it cannot process "missing legs" in a photo, all images with that specific occlusion will highlight the same set of malfunctioning filters. This allows for "debugging" via surgical fine-tuning: updating as few as 150 filters (~0.6% of a ResNet-50) on a single image can fix errors for that image and its semantic neighbors.
Experimental Results
The method was validated across ImageNet, CIFAR-10, and ImageNet-v2 using architectures like ResNet, VGG, and Inception.
Pruning and Error Correction
To validate the saliency maps, the authors performed pruning experiments. Turning off the top 100 most salient filters reduced incorrect class confidence by approximately 30%, significantly outperforming random pruning. More importantly, this pruning corrected the labels for 12% of misclassified ImageNet samples without any further training.
Spurious Correlations and NLP Extension
Case studies revealed that the model often relies on spurious correlations. For example, a shark might be misclassified due to background elements like boats. The parameter saliency maps successfully unmasked these dependencies. Preliminary tests on BERT also showed that the method identifies similar grammatical failure modes in language models, suggesting broad applicability.
Conclusion
Parameter-space saliency provides a robust framework for diagnosing and fixing neural network errors. By identifying the specific filters responsible for failures, researchers can move beyond "what" the model sees to "why" its internal logic fails. While challenges remain in managing the side effects of targeted fine-tuning, this approach opens new avenues for model debugging and reliability.
Summary updated