Introduction
Deep neural networks (DNNs) are often criticized as "black boxes" because their internal decision-making processes are difficult to interpret. While we can observe the final output, understanding how raw input is transformed into meaningful features layer-by-layer remains a significant challenge. This research introduces a mathematical framework to quantify and track the evolution of "knowledge" within a network, defining it through symbolic interaction primitives.
Defining Knowledge as Interaction Primitives
The core of this framework is the concept of AND/OR interactions. Instead of looking at individual neurons, the authors define knowledge as specific co-occurrence relationships between input variables.
- Interaction Primitives: An "AND" interaction represents a set of input patches (e.g., parts of a dog's face) that must all be present to contribute to the network's output.
- Complexity (Order): The "order" of an interaction is determined by the number of input variables involved. For instance, a 1-order interaction might be a simple patch of lawn, while a 7-order interaction represents a complex pattern like a dog's face.
To analyze intermediate layers, the researchers train linear probes on hidden features and extract these interactions. This allows them to track metrics such as Overlap (knowledge used in the final prediction), Forget (discarded early knowledge), and New (knowledge discovered in later layers).
The Two-Stage Process: Learning and Forgetting
A key discovery of this work is that DNNs typically follow a two-stage evolution of knowledge across their layers.
- Learning Phase: In early and middle layers, the network acts as a sponge. It fits target interactions but also encodes a high volume of redundant, noisy patterns. This is visible in the peak of lower-order interaction strengths in middle layers.
- Forgetting Phase: In later layers, the network acts as a filter. It prunes away redundant interactions, leaving behind stable, high-order primitives (like the dog's face) that are essential for the final classification.
Key Insights
- Simplicity and Generalization: Low-order interactions (simple relationships) are more likely to be shared across different models (e.g., ResNet vs. VGG) trained on the same task. In contrast, high-order interactions are often brittle, noise-sensitive, and unique to a specific model's overfitting.
- Knowledge Sparsity: Despite having millions of parameters, a network's actual "knowledge" is remarkably sparse. For example, a typical MNIST prediction is driven by only about 21 salient interactions.
- Stability: Low-order interactions remain stable under Gaussian noise, whereas complex, high-order interactions fluctuate significantly, suggesting they are less reliable for robust decision-making.
Experimental Validation
The framework was tested across multiple architectures (VGG, ResNet, MLP, BERT, XLNet) and datasets (MNIST, CIFAR-10, SST-2, tabular data). The results consistently showed that "completeness" of knowledge rises early, but "redundancy" peaks in middle layers before dropping sharply as the model refines its representation in the final layers.
Conclusion and Future Directions
This research provides a new lens through which to view the internal logic of DNNs. By quantifying knowledge as interaction primitives, we can see how networks balance fitting data with pruning noise. Future work will focus on applying this framework to the training process itself—observing how these interactions emerge over time—and developing methods to encourage models to prioritize stable, low-order interactions for better reliability and human alignment.
Summary updated