Rethinking the Role of Gradient-based Attribution Methods for Model Interpretability

Suraj Srinivas, François FleuretICLR 2021 · OpenReview

TL;DR

This paper reveals that the input-gradients we use to "explain" deep learning models are mathematically arbitrary and can be manipulated without changing the model's predictions. The authors show that these gradients actually represent an "implicit generative model" hidden within the classifier, and their interpretability depends on how well this hidden model aligns with the real-world data distribution.

What Problem Does This Paper Address?

In the field of Explainable AI (XAI), "saliency maps"—visualizations that highlight which pixels were most important for a model's decision—are a staple. Most of these methods rely on input-gradients: the derivative of the model's output with respect to the input image.

However, there is a fundamental theoretical problem: due to the "shift-invariance" of the softmax function, you can add a random function to a model's internal scores (logits) that completely changes the gradients while leaving the final classification probabilities identical. This raises a troubling question: if gradients can be anything, why do they usually look so structured and "human-readable" in standard models? The authors set out to find the hidden factor that gives these gradients their explanatory power.

Methods

The authors propose a shift in perspective. Instead of viewing logits simply as classification scores, they treat them as unnormalized log-densities of the data distribution. Under this view, input-gradients are actually "score functions" that point toward regions of higher data density.

To test this, they use a technique called Score-Matching. This is a generative modeling objective that forces the model's gradients to align with the gradients of the ground-truth data distribution. Because standard score-matching is computationally expensive (requiring second-order derivatives or "Hessians"), the authors introduce a novel Taylor Trace Estimator. This approximation allows them to train large-scale models using score-matching with only a single additional forward pass, avoiding the need for costly backward passes.

They compare four types of models (evaluated on CIFAR-10 and CIFAR-100 using 18-layer ResNets):

  1. Baseline ResNet: Standard training.
  2. Score-Matched: Trained to align its implicit density model with the data.
  3. Anti-Score-Matched: Intentionally trained to misalign the density model (as a control).
  4. Gradient-Norm Regularized: A common technique that penalizes large gradients.

Key Insights and Ideas

  • The "Implicit Density Model": Every softmax classifier contains a "secret" generative model. When we look at a saliency map, we aren't just seeing what the classifier "thinks" is important; we are seeing what this internal generative model thinks a "typical" image of that class looks like.
  • Gradients are Score Functions: The reason gradients look like the objects they classify (e.g., a "dog" gradient looking like a dog) is likely because the training process accidentally aligns the model's internal density with the real data distribution.
  • Interpretability is Generative, not Discriminative: The authors argue that tools like "activity maximization" (generating an image that maximizes a class score) are actually just biased sampling methods from this implicit density model.
  • Saliency is Fragile: Because gradients can be manipulated without changing the discriminative function, the authors warn that we should "re-think" using logit-gradients as a reliable window into a model's decision-making process.

Key Results

  • Quantitative Alignment: Using the "GAN-test" (measuring the accuracy of a separate classifier on images generated from the model's gradients), score-matched models performed significantly better (~73%) than the baseline (~59%), while anti-score-matched models plummeted (~16%).
  • Better Explanations: In pixel perturbation tests (where "important" pixels are removed to see how fast accuracy drops), gradients from score-matched models were much more effective at identifying truly influential pixels.
  • Visual Clarity: Qualitatively, the saliency maps and activity maximizations for score-matched models are much cleaner and more perceptually aligned with human intuition compared to the noisy gradients of baseline or anti-score-matched models.
Visual comparison of activity maximization across different training methods.
  • Surprising Baseline: Interestingly, simple gradient-norm regularization often performed as well as or better than explicit score-matching, suggesting it acts as a "proxy" for density alignment.

Limitations and Future Work

The authors acknowledge that while they've identified what makes gradients structured, they haven't fully explained why standard SGD training produces this alignment in the first place. They conjecture it might be due to implicit regularization inherent in the optimization process.

Additionally, score-matching can lead to a slight trade-off in classification accuracy (e.g., a drop from 78% to 72% in some tests). Future work could explore how to achieve high-quality density alignment without sacrificing discriminative performance. Finally, the authors suggest that practitioners should avoid logit-gradients for interpretability unless they are using models specifically regularized for density alignment.

Summary updated

Abstract

Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t. the inputs. The common assumption is that these input-gradients contain information regarding $p_{\theta} ( y\mid \mathbf{x} )$, the model's discriminative capabilities, thus justifying their use for interpretability. However, in this work, we show that these input-gradients can be arbitrarily manipulated as a consequence of the shift-invariance of softmax without changing the discriminative function. This leaves an open question: given that input-gradients can be arbitrary, why are they highly structured and explanatory in standard models? In this work, we re-interpret the logits of standard softmax-based classifiers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional generative model $p_{\theta}(\mathbf{x} \mid y)$ implicit in the discriminative model. This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model $p_{\theta}(\mathbf{x} \mid y)$ with that of the ground truth data distribution $p_{\text{data}} (\mathbf{x} \mid y)$. We test this hypothesis by studying the effect of density alignment on gradient explanations. To achieve this density alignment, we use an algorithm called score-matching, and propose novel approximations to this algorithm to enable training large-scale models. Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect. This also leads us to conjecture that unintended density alignment in standard neural network training may explain the highly structured nature of input-gradients observed in practice. Overall, our finding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.
BibTeX
@inproceedings{srinivas2021rethinking,
  author = {Suraj Srinivas and François Fleuret},
  title = {{Rethinking the Role of Gradient-based Attribution Methods for Model Interpretability}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=cPWSHJ7Om494},
}

Loading related content...

Figure