Methods for interpreting and understanding deep neural networks

Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller

Deep Neural Networks (DNNs) are the workhorses of modern AI, but their "black box" nature often leaves researchers asking: Why did the model do that? This paper, based on a tutorial from ICASSP 2017, provides a comprehensive framework for moving beyond predictive accuracy toward true model transparency.

TL;DR

This paper systematizes methods for "post-hoc" interpretability, distinguishing between global interpretations (what a model knows) and local explanations (why it made a specific decision). It highlights Layer-wise Relevance Propagation (LRP) and Deep Taylor Decomposition as robust frameworks for producing human-understandable heatmaps that reveal whether a model is truly learning features or just exploiting data artifacts.

What Problem Does This Paper Address?

As DNNs are deployed in high-stakes environments like medicine and autonomous driving, "high accuracy" is no longer a sufficient metric for trust. A model might achieve high scores by exploiting "artifacts"—irrelevant patterns in the data that correlate with the label but have no causal link (the "Clever Hans" effect).

The authors address the challenge of functional understanding: characterizing a model’s behavior without necessarily needing to decode every individual weight or neuron. They aim to bridge the gap between complex, non-linear mathematical functions and the interpretable domains humans understand, such as images, text, or physical properties.

Methods

The paper categorizes interpretability into two main branches:

  1. Interpreting the Model (Global):
    • Activation Maximization (AM): This technique searches for an input pattern that produces a maximum response for a specific class.
  2. The "Expert" Regularizer: To prevent AM from generating unrealistic noise, the authors discuss using a data density model (an "expert") to ensure the resulting prototypes look like real data (e.g., using RBMs or Generative Adversarial Networks).
  3. Explaining Decisions (Local):
    • Sensitivity Analysis: Uses the gradient of the output with respect to the input. It answers "what makes this image more like a car?" but often fails to explain the total prediction.
  4. Simple Taylor Decomposition: Decomposes the function value into a sum of relevance scores based on a first-order Taylor expansion.
  5. Layer-wise Relevance Propagation (LRP): The core contribution. LRP redistributes the prediction score from the output layer back through the network to the input pixels using specific conservation rules (like the $\alpha\beta$-rule). This ensures that the "total relevance" is preserved across layers.
  6. Deep Taylor Decomposition: Provides the formal theoretical grounding for LRP, viewing the redistribution process as a sequence of local Taylor expansions.

Datasets

The paper serves as a tutorial and uses several benchmark datasets to demonstrate these techniques:

  • MNIST: For visualizing digit prototypes and explanation continuity.
  • ILSVRC (ImageNet): Using architectures like BVLC CaffeNet and GoogleNet to show how LRP handles complex natural images.
  • PASCAL VOC 2012: Specifically used to demonstrate how models can be "right for the wrong reasons" (e.g., identifying a horse based on a copyright tag).
  • 20-Newsgroups: For text classification explanations.
  • Scientific Datasets: Including molecular structures (quantum chemistry), EEG brain recordings, and DNA sequences for protein binding.

Key Insights and Ideas

  • Interpretation vs. Explanation: The authors define interpretation as mapping an abstract concept (like "dog") into a domain humans understand, while explanation is identifying the specific features in a single instance that led to a decision.
  • The "Clever Hans" Trap: In a striking example, the authors show that a Fisher Vector classifier "correctly" identified horses only because it learned to detect a specific copyright tag present in many horse photos. LRP heatmaps made this failure immediately obvious, whereas standard accuracy metrics did not.
  • Continuity Matters: Sensitivity analysis and simple Taylor methods often produce "noisy" or discontinuous explanations. LRP is shown to be more stable, meaning nearly identical inputs yield nearly identical explanations.
  • Selectivity (Pixel-Flipping): To quantify "goodness," the authors use a "pixel-flipping" metric. By removing the most relevant pixels (as identified by the model) and measuring how fast the prediction score drops, they prove that LRP identifies truly influential features more accurately than gradient-based methods.

Key Results

  • Qualitative: LRP produces heatmaps that are significantly more "human-centric," highlighting the actual contours of objects rather than just edges or noise.
  • Quantitative: In selectivity tests (Area Under the Curve of the prediction score vs. features removed), LRP consistently outperforms sensitivity analysis, showing a much sharper drop in model confidence when "relevant" features are removed.
  • Scientific Discovery: The paper demonstrates that these methods can extract new insights from complex physical systems, such as identifying which atoms in a molecule contribute most to its atomization energy.

Limitations and Future Work

  • Hyperparameter Sensitivity: The choice of LRP rules (like $\alpha$ and $\beta$ values) can change the "sparsity" of the explanation. Choosing the right rule often requires empirical testing (e.g., via pixel-flipping).
  • Deep Fully Connected Layers: LRP tends to lose selectivity in very deep fully connected layers as relevance gets redistributed too broadly.
  • Data Manifold: The evaluation of explanations (removing features) can push the data off the "manifold" (making it look unlike real data), which might confuse the model in ways unrelated to the explanation's quality.
  • Mechanistic Gap: The paper focuses on functional understanding (black-box behavior) rather than mechanistic understanding (how internal representations are formed), leaving the latter as a separate, ongoing challenge in the field.

Summary updated

Abstract

This paper provides an entry point to the problem of interpreting a deep neural network model and explaining its predictions. It is based on a tutorial given at ICASSP 2017. As a tutorial paper, the set of methods covered here is not exhaustive, but sufficiently representative to discuss a number of questions in interpretability, technical challenges, and possible applications. The second part of the tutorial focuses on the recently proposed layer-wise relevance propagation (LRP) technique, for which we provide theory, recommendations, and tricks, to make most efficient use of it on real data.
BibTeX
@article{montavon2018methods,
  author = {Grégoire Montavon and Wojciech Samek and Klaus-Robert Müller},
  title = {{Methods for interpreting and understanding deep neural networks}},
  journal = {Digit. Signal Process.},
  year = {2018},
  doi = {10.1016/j.dsp.2017.10.011},
}

Loading related content...

Figure