Abstract
There is a growing interest in strategies that can help us understand or interpret neural networks—not merely providing a prediction, but offering context explaining why and how. While many current methods offer post-hoc analysis tools, recent results suggest that when classes map to high-level "concepts" in preceding layers, network behavior is easier to interpret. Such training can be accomplished via dynamic/EM routing where paths for individual classes are dynamic and sparse. In this paper, we show how a simple modification of the SGD scheme can provide dynamic/EM routing type behavior in convolutional neural networks (CNNs). Through extensive experiments, we evaluate the effect of this idea for interpretability, obtaining promising results while showing that no compromise in attainable accuracy is involved.
Introduction: The Black Box Problem
Deep learning models are notoriously "black boxes." While they achieve state-of-the-art accuracy, it is often impossible to discern why a model classified an image as a "dog" rather than a "cat." Most existing interpretability tools are post-hoc, attempting to explain a model after training. However, these explanations can be inconsistent or even misleading.
A more robust approach is to build models that are interpretable by design. Capsule Networks attempted this using "EM routing," forcing the network to choose specific paths for specific concepts. However, Capsule Networks are computationally expensive and difficult to scale. This research asks: Can we get the interpretability benefits of dynamic routing within standard, scalable CNN architectures?
Methodology: Dynamic-K Activation
The authors introduce Dynamic-K Activation and the resulting Dynamic Connection Layer. Instead of a standard fully connected layer where every input neuron connects to every output neuron, Dynamic-K forces the network to select only the top $K$ most relevant activations for each class.
The Mechanism
During the forward pass, for a given class (like "Airplane Cabin"), the layer identifies the $K$ units in the previous layer that contribute most strongly to that prediction and ignores the rest. This thresholding behavior is mathematically equivalent to a specific variant of Stochastic Gradient Descent (SGD) using Lipschitz sampling, providing theoretical guarantees that the model will converge as reliably as a standard CNN. Implementation is remarkably simple, requiring only 5-6 lines of code to modify the standard training routine.
Key Insights: Semantic Alignment and Sparsity
Traditionally, researchers feared that forcing a network to be sparse (using fewer connections) would hurt performance. This paper shows that by dynamically choosing which neurons to use based on the input, you can maintain—or even slightly improve—accuracy while being extremely sparse.
In a standard CNN, a "TV" prediction might be negatively associated with a "keyboard" simply because of how the math of cross-entropy loss works. In a Dynamic NN, the weights align better with human logic: a "Laptop" prediction is supported by "Keyboard" and "Screen" neurons, with no "repelling" effect between unrelated but co-occurring items. Furthermore, the model learns concept composability, representing complex scenes as a combination of a few simple concepts. For example, it might identify a "canal" using exactly four concepts: water, dam, castle, and sea.
Experimental Results
The researchers evaluated their approach on two primary benchmarks: CIFAR-10 and Places365. The latter is a complex dataset of 1.8 million images specifically chosen because its scene categories (like "stadium" or "park") are composed of clear semantic concepts (like "grass" or "trees").
- High Sparsity, High Accuracy: On the Places365 dataset, the "Res18-d5a" model (using only 5 active units) achieved 99% sparsity in its connections while staying within 0.1% of the accuracy of a dense, standard ResNet-18.
- Superior Interpretability: Using the NetDissect metric, Dynamic-K models consistently outperformed standard CNNs. The units were more concept-specific and less noisy.
- Cleaner Explanations: When visualizing why a model made a choice, the Dynamic-K model used fewer than 5 clear concepts to explain 98% of an image, whereas standard CNNs produced "spread out" explanations involving dozens of irrelevant features.
Conclusion and Future Work
This work demonstrates that interpretability does not have to come at the cost of performance or scalability. By replacing the final fully connected layer with a Dynamic Connection Layer, CNNs can achieve massive sparsity and human-aligned logic. Future work may involve extending this routing behavior further into the earlier layers of the network to see if concept-based learning can be enforced from the very first pixel, and further refining the selection of the hyperparameter $K$.
Summary updated