Adaptive Activation Thresholding: Dynamic Routing Type Behavior for Interpretability in Convolutional Neural Networks

Yiyou Sun, Sathya N. Ravi, Vikas SinghICCV 2019 · OpenReview

Abstract

There is a growing interest in strategies that can help us understand or interpret neural networks—not merely providing a prediction, but offering context explaining why and how. While many current methods offer post-hoc analysis tools, recent results suggest that when classes map to high-level "concepts" in preceding layers, network behavior is easier to interpret. Such training can be accomplished via dynamic/EM routing where paths for individual classes are dynamic and sparse. In this paper, we show how a simple modification of the SGD scheme can provide dynamic/EM routing type behavior in convolutional neural networks (CNNs). Through extensive experiments, we evaluate the effect of this idea for interpretability, obtaining promising results while showing that no compromise in attainable accuracy is involved.

Introduction: The Black Box Problem

Deep learning models are notoriously "black boxes." While they achieve state-of-the-art accuracy, it is often impossible to discern why a model classified an image as a "dog" rather than a "cat." Most existing interpretability tools are post-hoc, attempting to explain a model after training. However, these explanations can be inconsistent or even misleading.

A more robust approach is to build models that are interpretable by design. Capsule Networks attempted this using "EM routing," forcing the network to choose specific paths for specific concepts. However, Capsule Networks are computationally expensive and difficult to scale. This research asks: Can we get the interpretability benefits of dynamic routing within standard, scalable CNN architectures?

Methodology: Dynamic-K Activation

The authors introduce Dynamic-K Activation and the resulting Dynamic Connection Layer. Instead of a standard fully connected layer where every input neuron connects to every output neuron, Dynamic-K forces the network to select only the top $K$ most relevant activations for each class.

Figure 1: The Dynamic-K architecture routes specific semantic concepts (like water or amusement devices) to specific class predictions, creating a sparse and interpretable path.

The Mechanism

During the forward pass, for a given class (like "Airplane Cabin"), the layer identifies the $K$ units in the previous layer that contribute most strongly to that prediction and ignores the rest. This thresholding behavior is mathematically equivalent to a specific variant of Stochastic Gradient Descent (SGD) using Lipschitz sampling, providing theoretical guarantees that the model will converge as reliably as a standard CNN. Implementation is remarkably simple, requiring only 5-6 lines of code to modify the standard training routine.

Key Insights: Semantic Alignment and Sparsity

Traditionally, researchers feared that forcing a network to be sparse (using fewer connections) would hurt performance. This paper shows that by dynamically choosing which neurons to use based on the input, you can maintain—or even slightly improve—accuracy while being extremely sparse.

Figure 3: Comparison between Standard NN and Dynamic NN. Standard NNs often have "noisy" negative weights, whereas Dynamic NNs align weights with human logic (e.g., Laptop is supported by Keyboard and Screen).

In a standard CNN, a "TV" prediction might be negatively associated with a "keyboard" simply because of how the math of cross-entropy loss works. In a Dynamic NN, the weights align better with human logic: a "Laptop" prediction is supported by "Keyboard" and "Screen" neurons, with no "repelling" effect between unrelated but co-occurring items. Furthermore, the model learns concept composability, representing complex scenes as a combination of a few simple concepts. For example, it might identify a "canal" using exactly four concepts: water, dam, castle, and sea.

Experimental Results

The researchers evaluated their approach on two primary benchmarks: CIFAR-10 and Places365. The latter is a complex dataset of 1.8 million images specifically chosen because its scene categories (like "stadium" or "park") are composed of clear semantic concepts (like "grass" or "trees").

  • High Sparsity, High Accuracy: On the Places365 dataset, the "Res18-d5a" model (using only 5 active units) achieved 99% sparsity in its connections while staying within 0.1% of the accuracy of a dense, standard ResNet-18.
  • Superior Interpretability: Using the NetDissect metric, Dynamic-K models consistently outperformed standard CNNs. The units were more concept-specific and less noisy.
  • Cleaner Explanations: When visualizing why a model made a choice, the Dynamic-K model used fewer than 5 clear concepts to explain 98% of an image, whereas standard CNNs produced "spread out" explanations involving dozens of irrelevant features.

Conclusion and Future Work

This work demonstrates that interpretability does not have to come at the cost of performance or scalability. By replacing the final fully connected layer with a Dynamic Connection Layer, CNNs can achieve massive sparsity and human-aligned logic. Future work may involve extending this routing behavior further into the earlier layers of the network to see if concept-based learning can be enforced from the very first pixel, and further refining the selection of the hyperparameter $K$.

Summary updated

Abstract

There is a growing interest in strategies that can help us understand or interpret neural networks -- that is, not merely provide a prediction, but also offer additional context explaining why and how. While many current methods offer tools to perform this analysis for a given (trained) network post-hoc, recent results (especially on capsule networks) suggest that when classes map to a few high level ``concepts'' in the preceding layers of the network, the behavior of the network is easier to interpret or explain. Such training may be accomplished via dynamic/EM routing where the network ``routes'' for individual classes (or subsets of images) are dynamic and involve few nodes even if the full network may not be sparse. In this paper, we show how a simple modification of the SGD scheme can help provide dynamic/EM routing type behavior in convolutional neural networks. Through extensive experiments, we evaluate the effect of this idea for interpretability where we obtain promising results, while also showing that no compromise in attainable accuracy is involved. Further, we show that the minor modification is seemingly ad-hoc, the new algorithm can be analyzed by an approximate method which provably matches known rates for SGD.
BibTeX
@inproceedings{sun2019adaptive,
  author = {Yiyou Sun and Sathya N. Ravi and Vikas Singh},
  title = {{Adaptive Activation Thresholding: Dynamic Routing Type Behavior for Interpretability in Convolutional Neural Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=L-RNLaIKHo0},
}
Figure