Influence-Directed Explanations for Deep Convolutional Networks

Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, Linyi Li

Abstract

Deep learning models are often criticized as "black boxes," providing little insight into their internal decision-making processes. This paper introduces influence-directed explanations, a framework that identifies which internal neurons most strongly drive a network's behavior across entire classes of data. By combining axiomatic mathematical foundations with internal layer analysis, the authors demonstrate that a tiny fraction of a network's neurons—the "essence" of a class—is sufficient to distinguish categories with high recall, often outperforming the original full model.

Introduction

Understanding deep neural networks typically involves two disparate approaches: feature attribution (saliency maps for specific images) and feature visualization (understanding what individual neurons detect). However, a gap remains for explanations that are both causal (identifying drivers of output) and generalizable (explaining behavior across a whole category). This research bridges that gap by identifying influential internal neurons that represent consistent concepts across a distribution of images.

Methodology: Distributional Influence

The core of this framework is Distributional Influence, a measure that identifies influential neurons by "slicing" the network at a specific layer. Unlike point-specific methods, this approach calculates the average partial derivative of a "quantity of interest" (e.g., the score for a specific class) over a "distribution of interest" (e.g., all images of that class).

This measure is built on an axiomatic foundation, satisfying mathematical properties such as Slice Invariance—ensuring the explanation remains consistent regardless of how layers are mathematically partitioned—and Linear Agreement. By adjusting the quantity of interest, researchers can even generate comparative explanations, such as identifying the specific features that distinguish a "sports car" from a "convertible."

Key Insights and Results

Generalization of Internal Influence

While the specific pixels that trigger a prediction vary between individual images, the internal concepts (like a "fur" or "ear" detector) remain consistent across a class. The authors found that measuring influence at internal layers yields explanations that hold true for the entire category.

Performance degradation curves showing that removing features based on mean internal influence (solid red) is as impactful as removing them based on individual image influence (dashed red).

As shown in the feature removal analysis above, when features are removed based on mean influence across a class, the network's performance drops almost as rapidly as when features are removed based on individual image influence. This confirms that influential internal neurons are truly representative of the entire class distribution.

Model Compression and "Class Experts"

One of the most striking findings is that less than 1% of a network's neurons are responsible for the "essence" of a class. To prove this, the authors created "experts"—mini-classifiers using only the top influential neurons for a specific class. These experts frequently achieved significantly higher recall than the original full model.

Recall metrics comparing the original model (Orig.), influence-based experts (Infl.), and activation-based experts (Act.).

For example, as seen in the table above, the 'Pelican' expert achieved 0.95 recall compared to the original model's 0.65, despite using only a fraction of the total neurons. Furthermore, the data highlights that Activation ≠ Influence: many highly active neurons (the 'Act.' column) have significantly less impact on the final prediction than those identified through influence gradients.

Conclusion and Future Work

Influence-directed explanations provide a powerful tool for peering into the black box of CNNs, offering both qualitative localization and quantitative performance gains through model compression. While currently focused on Convolutional Neural Networks for image tasks, future work aims to extend this framework to recursive networks and develop comprehensive debugging suites to identify systematic model biases or errors.

Summary updated

BibTeX
@inproceedings{leino2018influence,
  author = {Klas Leino and Shayak Sen and Anupam Datta and Matt Fredrikson and Linyi Li},
  title = {{Influence-Directed Explanations for Deep Convolutional Networks}},
  booktitle = {ITC},
  year = {2018},
  doi = {10.1109/test.2018.8624792},
}
Figure