Abstract
Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t. the inputs. The common assumption is that these input-gradients contain information regarding $p_{ heta} ( y \mid x)$, the model's discriminative capabilities, thus justifying their use for interpretability. However, in this work we show that these input-gradients can be arbitrarily manipulated as a consequence of the shift-invariance of softmax without changing the discriminative function. This leaves an open question: if input-gradients can be arbitrary, why are they highly structured and explanatory in standard models? We investigate this by re-interpreting the logits of standard softmax-based classifiers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional density model $p_{ heta}(x \mid y)$ implicit within the discriminative model. This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model $p_{ heta}(x \mid y)$ with that of the ground truth data distribution $p_{\text{data}} (x \mid y)$. We test this hypothesis by studying the effect of density alignment on gradient explanations. To achieve this alignment we use score-matching, and propose novel approximations to this algorithm to enable training large-scale models. Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect. Overall, our finding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.
Introduction: The Fragility of Saliency Maps
Saliency maps, generated by taking the gradient of a model's output with respect to its input, are a cornerstone of AI interpretability. They are intended to highlight the features (like a cat's ears or whiskers) that drive a classification decision. However, a fundamental theoretical issue exists: the shift-invariance of softmax. Because adding an arbitrary function to the logits does not change the final prediction but does change the input gradients, these explanations can be mathematically manipulated to look like anything—even pure noise—without affecting model accuracy.
This raises a critical question: if gradients are arbitrary, why do standard models produce such structured and human-readable explanations?
The Implicit Generative Model
The authors propose that a classifier is not merely a discriminative tool but also contains an implicit generative model. By interpreting the logits as unnormalized log-densities, the input gradient becomes the score function (the gradient of the log-density) of this internal model. The "interpretability" of gradients is thus a measure of how well this internal density model aligns with the actual data distribution $p_{\text{data}}(x|y)$.
To test this, the researchers employed Score-Matching, an objective that explicitly aligns the model's internal score with the ground-truth distribution.
Scaling Score-Matching: The Taylor Trace Estimator
Standard score-matching is computationally expensive because it requires calculating the Trace of the Hessian. To make this feasible for large architectures like ResNets, the authors introduced the Taylor Trace Estimator (TTE). This innovation approximates the Hessian trace using only a single additional forward pass, enabling the training of deep models with density alignment objectives.
Key Results: Alignment Drives Interpretability
The experiments on CIFAR-10 and CIFAR-100 confirmed that density alignment is the primary driver of gradient structure.
- Qualitative Clarity: As shown in Figure 6, models trained with score-matching (d) produce significantly more coherent and perceptually aligned images via gradient ascent compared to the baseline (a). Conversely, "Anti-score-matching" (b), which pushes the model away from the data distribution, results in nonsensical gradients.
- The GAN-test: When a separate classifier was used to identify images "generated" by the models, the score-matched model achieved 72.75% accuracy, far outperforming the baseline's 59.47%.
- Quantitative Evaluation: Pixel perturbation tests showed that score-matched gradients are more effective at identifying truly influential features for the model's performance.
Conclusion
This research shifts our understanding of saliency maps. Rather than purely explaining a classification decision, input gradients reflect the model's internal "map" of the data distribution. While standard training (SGD) may provide some unintended alignment, explicit score-matching can significantly enhance the reliability and clarity of model explanations, albeit with a small trade-off in classification accuracy.
Summary updated