Abstract
Generalization—the ability of a neural network to perform on unseen data—remains one of the most elusive phenomena in machine learning. This paper introduces GenEFT (Generalization Effective Theory), a physics-inspired framework that treats generalization as a physical system. By combining information-theoretic "statics" with a "many-body" particle model of latent representations called "dynamics," GenEFT provides a predictive tool for determining the minimum data required for generalization and the specific learning rate regimes that prevent a model from falling into the trap of memorization.
Introduction
While generalization is the fundamental objective of machine learning, predicting when and why it occurs is notoriously difficult. Theoretical bounds are often too abstract for practical application, while empirical "rules of thumb" lack a rigorous mathematical foundation. GenEFT seeks to bridge this gap by providing a unified framework to understand both the statics (data requirements) and the dynamics (optimization trajectories) of generalization.
The GenEFT Framework
The authors split the problem of generalization into two distinct but related components: statics and dynamics.
1. Statics: Information Theory and Description Length
In the static view, a learning task is modeled as a problem of "description length." By calculating the number of bits required to define a specific mathematical relationship (such as "greater than" or "equal modulo 3"), the authors estimate the minimum amount of data needed to specify that relationship. Using a "balls-in-buckets" approximation, they predict the phase transition where a model moves from random guessing to true knowledge.
2. Dynamics: Interacting Repon Theory
The most novel contribution of GenEFT is the treatment of latent space representations as interacting particles called "repons." By analyzing the gradient flow of the loss function, the authors derive equations of motion for these repons that mirror classical physics. They identify a conserved quantity—analogous to energy in a harmonic oscillator—that determines the model's fate: whether repons will "collide" (leading to generalization) or "repel" (leading to memorization and overfitting).
Experimental Methodology
The researchers validated GenEFT using knowledge graph learning within an autoencoder architecture. They tested four primary synthetic relations on a set of $n=30$ elements:
- Modulo 3 & Modulo 5: Testing equivalence classes (symmetry and transitivity).
- Greater-than: Testing total ordering (antisymmetry and transitivity).
- Complete Bipartite Graphs: Testing anti-transitive structures.
These datasets were selected because their description lengths are mathematically well-defined, allowing for direct comparison between theoretical predictions and experimental results.
Key Insights and Results
The Inductive Gap and Predictive Power
The information-theoretic model accurately predicted the "critical training fraction" ($p_c$) where models begin to generalize. However, the authors observed an "inductive gap"—extra data bits spent by the model to learn the correct inductive bias (e.g., discovering that a relationship is symmetric). Simplifying the decoder architecture to match the task's mathematical structure was shown to significantly reduce this gap.
The 10x Rule for Learning Rates
GenEFT reveals a mathematical duality between many-body physics and representation learning. A critical finding is that the ratio of the decoder learning rate to the encoder learning rate ($\eta_{dec}/\eta_{enc}$) is a primary determinant of success. Empirical results suggest a "10x rule": for generalization to occur, this ratio should generally remain below 10. If the decoder learns too quickly, the repons fail to converge at the correct points in latent space, resulting in memorization.
Phase Boundaries
The authors produced phase diagrams showing clear boundaries between Generalization, Grokking (delayed generalization), and Overfitting. Interestingly, these boundaries appear largely independent of the specific task, depending instead on the learning rate ratios.
Conclusion and Future Work
GenEFT provides a powerful new lens for understanding generalization, moving beyond empirical observation toward predictive theory. While currently demonstrated on synthetic graph relations, the framework offers a path toward understanding more complex architectures. Future work will focus on extending the "repon" analogy to high-dimensional, real-world data like natural language and images, and exploring the "Goldilocks Zone" of optimal decoder complexity.
Summary updated