Deep neural networks (DNNs) are often criticized as "black boxes" because their decision-making logic is buried within millions of parameters. However, a growing body of research led by Quanshi Zhang and colleagues suggests that we can decompose this logic into symbolic "interactive concepts." By analyzing the complexity of these concepts—specifically their interaction order—we can finally explain why models generalize, why they overfit, and why they are vulnerable to adversarial attacks.
The Building Blocks of Model Logic
Instead of looking at individual pixels or neurons, researchers like Mingjie Li and Quanshi Zhang in Does a Neural Network Really Encode Symbolic Concepts? propose viewing model outputs as a sum of "Harsanyi interactions."
An interaction is essentially a logical AND relationship between a subset of input variables. For example, a model might recognize a "face" only when "eyes," "nose," and "mouth" are all present. The order of an interaction is simply the number of variables involved.
- Low-order interactions represent simple, local patterns (e.g., a single edge or a small texture).
- High-order interactions represent complex, global structures (e.g., the precise alignment of dozens of features that define a specific training image).
Why Complexity Breeds Instability
The central discovery across this research area is that the "order" of an interaction dictates its stability. Huilin Zhou, Dongrui Liu, and others have proven that the sensitivity of a concept to noise grows exponentially with its interaction order.
In Explaining Generalization Power of a DNN Using Interactive Concepts, the authors demonstrate that high-order interactions require the precise coordination of many variables. If even one variable is slightly perturbed by noise, the entire high-order interaction can collapse or fluctuate wildly. Mathematically, this means high-order concepts have much higher variance than low-order ones.
This exponential relationship explains the fundamental trade-off in machine learning:
- Generalization: Low-order interactions are stable and consistent across different datasets. Models that rely on these "simple" rules generalize better to unseen data (Huilin Zhou et al.).
- Overfitting: High-order interactions allow a model to fit complex training data perfectly, but because they are so sensitive to noise, they rarely transfer to the test set. They represent the model "memorizing" specific training samples rather than learning general rules.
The Dynamics of Learning and Forgetting
How does a model decide which order of interactions to use? Junpeng Zhang and Quanshi Zhang identified a "two-phase" dynamic in Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features.
Initially, models focus on stripping away noise to find simple, low-order interactions. However, as training continues, they enter a "creeping complexity" phase where they begin to learn increasingly high-order interactions to minimize training error. This is the exact moment overfitting begins. Interestingly, Xu Cheng found in Layerwise Change of Knowledge in Neural Networks that deeper layers in a network often act as filters, "forgetting" these brittle high-order interactions to settle on more stable, generalizable knowledge.
Robustness and the Interaction Bottleneck
This framework also provides a "grand unified theory" for adversarial transferability. In Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferability, Quanshi Zhang and his team showed that adversarial attacks are most effective across different models when they target low-order interactions. High-order interactions are too model-specific; an attack that exploits them will "overfit" to the source model and fail to fool a target model.
To defend against such attacks, we can bias models toward simplicity. Qihan Ren proved in Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Concepts that Bayesian Neural Networks (BNNs) are naturally robust because weight uncertainty makes high-order interactions too volatile to survive. The BNN is forced to rely on low-order concepts that are stable even when weights shift.
Finally, Huiqi Deng and Qihan Ren recently discovered an "Interaction Bottleneck" (The Interaction Bottleneck of Deep Neural Networks). They found that DNNs struggle to represent interactions of "middle" complexity. They tend to favor either very simple local features or very complex global structures. By modulating this bottleneck, researchers can now explicitly trade off a model's ability to fit complex data against its inherent robustness.