Understanding a Deep Neural Network Based on Neural-Path Coding

Dawei Dai, Chengfu Tang, Dongyao Zhao, Shuyin Xia, Haoyue BaiIEEE Access 2020 · OpenReview

Abstract

Deep neural networks (DNNs) are often viewed as "black boxes" because their internal logic is unpredictable and difficult to interpret. This research introduces the concept of "neural-path coding," inspired by biological neural circuits, to quantify and control how information flows through a network. By measuring the Euclidean distance (ED) between the neural-paths of different classes, the researchers provide a geometric explanation for model confusion: the shorter the distance between paths, the more likely the model is to make mistakes. Furthermore, the study proposes a "controlled learning process" using redirect layers to pre-assign neurons to specific classes, creating more interpretable architectures that maintain high accuracy.

Introduction

Despite the massive success of Deep Neural Networks, they remain notoriously opaque. In high-stakes fields like medicine or autonomous driving, the inability to explain why a network makes a specific decision is a significant barrier to trust. This paper addresses three fundamental questions: What does a network learn? Why is that knowledge unpredictable? And why are certain classes easily confused?

Inspired by neuroscience—where specific neural circuits in the brain encode specific decisions—the authors hypothesize that DNNs develop similar "neural-paths." By identifying and controlling these paths, we can move away from the black-box model toward a more "expectable" and transparent architecture.

The Neural-Path Hypothesis

The core idea is that information is transmitted between neurons along specific circuits to reach end neurons that encode a decision. In a standard training process, these paths often overlap, with neurons shared between multiple classes. This sharing is a primary source of unpredictability and error.

Comparison of standard and controlled learning processes. (a) Initial network, (b) standard learning with shared neurons, and (c) controlled learning with distinct paths.

As shown in Figure 1, the "controlled learning process" aims to separate these paths, ensuring that specific neurons are dedicated to specific classes. This makes the internal logic of the network visible and predictable.

Methodology: Quantifying and Controlling Information Flow

The researchers developed a three-step technical framework to implement this hypothesis:

  1. Neural-Path Quantification: For a trained model, a "neural-path" is defined by identifying neurons or feature maps that show high average activation levels for a specific class. This allows the researchers to map the flow of information through hidden layers.
  2. Distance Metric: By defining a Euclidean Distance (ED) between the neural-paths of any two classes, the team can mathematically measure how similar the network's internal representations are.
  3. Redirect Layers: To build interpretability by design, the authors introduced "redirect layers." In these layers, specific feature maps are pre-assigned to specific classes. Custom loss functions ($loss_{pass}$ to encourage activation for the assigned class and $loss_{forbid}$ to suppress it for others) force the network to form distinct, class-specific lanes.
A specific neural-path through hidden layers $l_{k-2}$, $l_{k-1}$, and $l_k$ for Class 1.

Results: The Geometry of Mistakes

Experiments conducted on MNIST, CIFAR-10/100, and ImageNet32 revealed a striking correlation: Mistakes are geometric. The inverse relationship between neural-path distance and error rate held true across various architectures, including MLPs, ResNets, and DenseNets.

A key finding, dubbed the "20/50 Rule," showed that the top 20% of class pairs with the smallest neural-path distances accounted for 50% to 60% of all incorrect predictions. This confirms that path proximity is a direct predictor of model confusion.

Crucially, the "controlled learning" approach proved that interpretability does not have to come at the cost of performance. In a "Dogs vs. Cats" experiment using a ResNet-50, the controlled model achieved 97.92% accuracy, nearly identical to the standard baseline of 98.02%. Visualization of these redirect layers provided a clear diagnostic tool: when the model failed, the "wrong" class-specific neurons were visibly firing.

Conclusion

By quantifying the internal circuits of a neural network, this research provides a path toward explainable AI. While scaling redirect layers to thousands of classes remains a challenge for future work, the ability to predict errors through geometric distance and steer the learning process marks a significant step toward making deep learning models truly understandable and trustworthy.

Summary updated

Abstract

Recently, the deep neural network has achieved considerable success in the field of machine learning. However, for most tasks, the current neural network models are still considered as the “black-box” structure, essentially because the knowledge a neural network learns from data is often unpredictable and unexplainable. Similar to neural coding in the brain, one neuron may be simultaneously involved in encoding one or even several tasks. Information may be transmitted between neurons along a particular neural-path (neural circuit) to end neurons that encode a specific decision. Inspired by this, we aim to explore the reason for the performance of a neural model based on a neural-path. First, for a trained neural model, we quantify the neural-path in which the role of a neuron was assumed to control the amount of information that can be passed through; Second, we define a Euclidean distance (ED) for every two neural-paths, and by analyzing the EDs between two classes of a neural classifier, we explain the ease of prediction for some classes, whereas others are not. We performed extensive experiments for architectures of ResNet and DenseNet models on several benchmark datasets, and determined that the shorter the distance between the neural-paths of two classes, the easier it is to make mistakes. Finally, we proposed a method for controlling the formation of a “neural-path” to build a partially understandable neural model. For the new ResNet model, each feature map in redirect layers was assigned to participate in encoding only one class. The feasibility of the method is also verified through experiments.
BibTeX
@inproceedings{dai2020understanding,
  author = {Dawei Dai and Chengfu Tang and Dongyao Zhao and Shuyin Xia and Haoyue Bai},
  title = {{Understanding a Deep Neural Network Based on Neural-Path Coding}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=q4Y2DmBshx},
}

Loading related content...

Figure