TL;DR
This paper reveals that the input-gradients we use to "explain" deep learning models are mathematically arbitrary and can be manipulated without changing the model's predictions. The authors show that these gradients actually represent an "implicit generative model" hidden within the classifier, and their interpretability depends on how well this hidden model aligns with the real-world data distribution.
What Problem Does This Paper Address?
In the field of Explainable AI (XAI), "saliency maps"—visualizations that highlight which pixels were most important for a model's decision—are a staple. Most of these methods rely on input-gradients: the derivative of the model's output with respect to the input image.
However, there is a fundamental theoretical problem: due to the "shift-invariance" of the softmax function, you can add a random function to a model's internal scores (logits) that completely changes the gradients while leaving the final classification probabilities identical. This raises a troubling question: if gradients can be anything, why do they usually look so structured and "human-readable" in standard models? The authors set out to find the hidden factor that gives these gradients their explanatory power.
Methods
The authors propose a shift in perspective. Instead of viewing logits simply as classification scores, they treat them as unnormalized log-densities of the data distribution. Under this view, input-gradients are actually "score functions" that point toward regions of higher data density.
To test this, they use a technique called Score-Matching. This is a generative modeling objective that forces the model's gradients to align with the gradients of the ground-truth data distribution. Because standard score-matching is computationally expensive (requiring second-order derivatives or "Hessians"), the authors introduce a novel Taylor Trace Estimator. This approximation allows them to train large-scale models using score-matching with only a single additional forward pass, avoiding the need for costly backward passes.
They compare four types of models (evaluated on CIFAR-10 and CIFAR-100 using 18-layer ResNets):
- Baseline ResNet: Standard training.
- Score-Matched: Trained to align its implicit density model with the data.
- Anti-Score-Matched: Intentionally trained to misalign the density model (as a control).
- Gradient-Norm Regularized: A common technique that penalizes large gradients.
Key Insights and Ideas
- The "Implicit Density Model": Every softmax classifier contains a "secret" generative model. When we look at a saliency map, we aren't just seeing what the classifier "thinks" is important; we are seeing what this internal generative model thinks a "typical" image of that class looks like.
- Gradients are Score Functions: The reason gradients look like the objects they classify (e.g., a "dog" gradient looking like a dog) is likely because the training process accidentally aligns the model's internal density with the real data distribution.
- Interpretability is Generative, not Discriminative: The authors argue that tools like "activity maximization" (generating an image that maximizes a class score) are actually just biased sampling methods from this implicit density model.
- Saliency is Fragile: Because gradients can be manipulated without changing the discriminative function, the authors warn that we should "re-think" using logit-gradients as a reliable window into a model's decision-making process.
Key Results
- Quantitative Alignment: Using the "GAN-test" (measuring the accuracy of a separate classifier on images generated from the model's gradients), score-matched models performed significantly better (~73%) than the baseline (~59%), while anti-score-matched models plummeted (~16%).
- Better Explanations: In pixel perturbation tests (where "important" pixels are removed to see how fast accuracy drops), gradients from score-matched models were much more effective at identifying truly influential pixels.
- Visual Clarity: Qualitatively, the saliency maps and activity maximizations for score-matched models are much cleaner and more perceptually aligned with human intuition compared to the noisy gradients of baseline or anti-score-matched models.
- Surprising Baseline: Interestingly, simple gradient-norm regularization often performed as well as or better than explicit score-matching, suggesting it acts as a "proxy" for density alignment.
Limitations and Future Work
The authors acknowledge that while they've identified what makes gradients structured, they haven't fully explained why standard SGD training produces this alignment in the first place. They conjecture it might be due to implicit regularization inherent in the optimization process.
Additionally, score-matching can lead to a slight trade-off in classification accuracy (e.g., a drop from 78% to 72% in some tests). Future work could explore how to achieve high-quality density alignment without sacrificing discriminative performance. Finally, the authors suggest that practitioners should avoid logit-gradients for interpretability unless they are using models specifically regularized for density alignment.
Summary updated