Layerwise Change of Knowledge in Neural Networks

Xu Cheng, Lei Cheng, Zhaoran Peng, Yang Xu, Tian Han, Quanshi Zhang

Introduction

Deep neural networks (DNNs) are often criticized as "black boxes" because their internal decision-making processes are difficult to interpret. While we can observe the final output, understanding how raw input is transformed into meaningful features layer-by-layer remains a significant challenge. This research introduces a mathematical framework to quantify and track the evolution of "knowledge" within a network, defining it through symbolic interaction primitives.

Overview of the framework: extracting interaction primitives from DNN layers to track the learning and forgetting phases.

Defining Knowledge as Interaction Primitives

The core of this framework is the concept of AND/OR interactions. Instead of looking at individual neurons, the authors define knowledge as specific co-occurrence relationships between input variables.

  • Interaction Primitives: An "AND" interaction represents a set of input patches (e.g., parts of a dog's face) that must all be present to contribute to the network's output.
  • Complexity (Order): The "order" of an interaction is determined by the number of input variables involved. For instance, a 1-order interaction might be a simple patch of lawn, while a 7-order interaction represents a complex pattern like a dog's face.

To analyze intermediate layers, the researchers train linear probes on hidden features and extract these interactions. This allows them to track metrics such as Overlap (knowledge used in the final prediction), Forget (discarded early knowledge), and New (knowledge discovered in later layers).

The Two-Stage Process: Learning and Forgetting

A key discovery of this work is that DNNs typically follow a two-stage evolution of knowledge across their layers.

The learning strength of different interaction orders across early, middle, and later layers, highlighting the transition from learning to forgetting.
  1. Learning Phase: In early and middle layers, the network acts as a sponge. It fits target interactions but also encodes a high volume of redundant, noisy patterns. This is visible in the peak of lower-order interaction strengths in middle layers.
  2. Forgetting Phase: In later layers, the network acts as a filter. It prunes away redundant interactions, leaving behind stable, high-order primitives (like the dog's face) that are essential for the final classification.

Key Insights

  • Simplicity and Generalization: Low-order interactions (simple relationships) are more likely to be shared across different models (e.g., ResNet vs. VGG) trained on the same task. In contrast, high-order interactions are often brittle, noise-sensitive, and unique to a specific model's overfitting.
  • Knowledge Sparsity: Despite having millions of parameters, a network's actual "knowledge" is remarkably sparse. For example, a typical MNIST prediction is driven by only about 21 salient interactions.
  • Stability: Low-order interactions remain stable under Gaussian noise, whereas complex, high-order interactions fluctuate significantly, suggesting they are less reliable for robust decision-making.

Experimental Validation

The framework was tested across multiple architectures (VGG, ResNet, MLP, BERT, XLNet) and datasets (MNIST, CIFAR-10, SST-2, tabular data). The results consistently showed that "completeness" of knowledge rises early, but "redundancy" peaks in middle layers before dropping sharply as the model refines its representation in the final layers.

Conclusion and Future Directions

This research provides a new lens through which to view the internal logic of DNNs. By quantifying knowledge as interaction primitives, we can see how networks balance fitting data with pruning noise. Future work will focus on applying this framework to the training process itself—observing how these interactions emerge over time—and developing methods to encourage models to prioritize stable, low-order interactions for better reliability and human alignment.

Summary updated

BibTeX
@inproceedings{cheng2024layerwise,
  author = {Xu Cheng and Lei Cheng and Zhaoran Peng and Yang Xu and Tian Han and Quanshi Zhang},
  title = {{Layerwise Change of Knowledge in Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=7zEoinErzQ},
}
Figure