Abstract
This paper provides an entry point to the problem of interpreting a deep neural network model and explaining its predictions. It is based on a tutorial given at ICASSP 2017. It introduces some recently proposed techniques of interpretation, along with theory, tricks and recommendations, to make most efficient use of these techniques on real data. It also discusses a number of practical applications.
Introduction
Deep Neural Networks (DNNs) have achieved remarkable success across diverse domains, yet their internal decision-making processes often remain opaque. As these models are increasingly deployed in critical sectors such as healthcare and autonomous systems, the need for transparency becomes paramount. High predictive accuracy is no longer sufficient; we must ensure models are "right for the right reasons" and not merely exploiting dataset artifacts—a phenomenon known as the "Clever Hans" effect. This paper systematizes methods for post-hoc interpretability, distinguishing between global model interpretation and local decision explanation.
Global Interpretation: Understanding Model Knowledge
Global interpretation aims to characterize what a model has learned about a specific class or concept. A primary technique is Activation Maximization (AM), which searches for an input pattern that maximizes the response of a specific output neuron.
The Role of the "Expert" Regularizer
Standard AM often results in "artificial-looking" prototypes dominated by high-frequency noise that the model has learned to associate with a class but which lacks human-interpretable structure. To mitigate this, the authors propose incorporating a data density model, or an "expert" $p(x)$, as a regularizer. This ensures that the generated prototypes reside on the data manifold, making them recognizable to humans.
The evolution of these architectures—from simple AM to those incorporating generative decoders—allows for the extraction of increasingly clear and representative prototypes, as demonstrated on the MNIST dataset.
Local Explanation: Explaining Individual Decisions
While global interpretation looks at the model's general knowledge, local explanation seeks to identify why a specific input led to a particular prediction.
Layer-wise Relevance Propagation (LRP)
The core contribution discussed is Layer-wise Relevance Propagation (LRP). Unlike sensitivity analysis, which relies on local gradients and often produces noisy heatmaps, LRP redistributes the total prediction score from the output layer back to the input pixels. This process follows conservation rules (such as the $\alpha\beta$-rule), ensuring that the "total relevance" is preserved across layers.
Theoretical Foundation: Deep Taylor Decomposition
LRP is formally grounded in Deep Taylor Decomposition, which views the redistribution process as a sequence of local first-order Taylor expansions. This framework provides a principled way to decompose a complex non-linear function into a sum of relevance scores for individual input features, resulting in stable and human-centric heatmaps.
Empirical Validation and the "Clever Hans" Trap
The utility of these methods is highlighted through the detection of the "Clever Hans" effect. In one instance, a high-performing classifier for the PASCAL VOC dataset was found to identify "horse" images primarily by detecting a specific copyright tag present in the training data. LRP heatmaps made this failure immediately apparent, whereas standard performance metrics did not.
To quantitatively evaluate explanations, the authors utilize a "pixel-flipping" metric. By iteratively removing the most relevant pixels and measuring the decline in the model's prediction score, they demonstrate that LRP identifies truly influential features more accurately than traditional gradient-based methods.
Conclusion and Future Directions
Interpreting DNNs is essential for building trust and ensuring scientific validity. While techniques like LRP and AM with expert regularizers provide powerful tools for functional understanding, challenges remain. Future work must address the sensitivity of these methods to hyperparameters and bridge the gap toward a more mechanistic understanding of internal neural representations. Nonetheless, these methods already enable significant insights in fields ranging from computer vision to quantum chemistry.
Summary updated