GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory

David D. Baek, Ziming Liu, Max Tegmark

Abstract

Generalization—the ability of a neural network to perform on unseen data—remains one of the most elusive phenomena in machine learning. This paper introduces GenEFT (Generalization Effective Theory), a physics-inspired framework that treats generalization as a physical system. By combining information-theoretic "statics" with a "many-body" particle model of latent representations called "dynamics," GenEFT provides a predictive tool for determining the minimum data required for generalization and the specific learning rate regimes that prevent a model from falling into the trap of memorization.

Introduction

While generalization is the fundamental objective of machine learning, predicting when and why it occurs is notoriously difficult. Theoretical bounds are often too abstract for practical application, while empirical "rules of thumb" lack a rigorous mathematical foundation. GenEFT seeks to bridge this gap by providing a unified framework to understand both the statics (data requirements) and the dynamics (optimization trajectories) of generalization.

The GenEFT Framework

The authors split the problem of generalization into two distinct but related components: statics and dynamics.

The GenEFT framework splits generalization into Statics (Information Theory) and Dynamics (Interacting Repon Theory).

1. Statics: Information Theory and Description Length

In the static view, a learning task is modeled as a problem of "description length." By calculating the number of bits required to define a specific mathematical relationship (such as "greater than" or "equal modulo 3"), the authors estimate the minimum amount of data needed to specify that relationship. Using a "balls-in-buckets" approximation, they predict the phase transition where a model moves from random guessing to true knowledge.

2. Dynamics: Interacting Repon Theory

The most novel contribution of GenEFT is the treatment of latent space representations as interacting particles called "repons." By analyzing the gradient flow of the loss function, the authors derive equations of motion for these repons that mirror classical physics. They identify a conserved quantity—analogous to energy in a harmonic oscillator—that determines the model's fate: whether repons will "collide" (leading to generalization) or "repel" (leading to memorization and overfitting).

Experimental Methodology

The researchers validated GenEFT using knowledge graph learning within an autoencoder architecture. They tested four primary synthetic relations on a set of $n=30$ elements:

  • Modulo 3 & Modulo 5: Testing equivalence classes (symmetry and transitivity).
  • Greater-than: Testing total ordering (antisymmetry and transitivity).
  • Complete Bipartite Graphs: Testing anti-transitive structures.

These datasets were selected because their description lengths are mathematically well-defined, allowing for direct comparison between theoretical predictions and experimental results.

Key Insights and Results

The Inductive Gap and Predictive Power

The information-theoretic model accurately predicted the "critical training fraction" ($p_c$) where models begin to generalize. However, the authors observed an "inductive gap"—extra data bits spent by the model to learn the correct inductive bias (e.g., discovering that a relationship is symmetric). Simplifying the decoder architecture to match the task's mathematical structure was shown to significantly reduce this gap.

The 10x Rule for Learning Rates

GenEFT reveals a mathematical duality between many-body physics and representation learning. A critical finding is that the ratio of the decoder learning rate to the encoder learning rate ($\eta_{dec}/\eta_{enc}$) is a primary determinant of success. Empirical results suggest a "10x rule": for generalization to occur, this ratio should generally remain below 10. If the decoder learns too quickly, the repons fail to converge at the correct points in latent space, resulting in memorization.

Phase Boundaries

The authors produced phase diagrams showing clear boundaries between Generalization, Grokking (delayed generalization), and Overfitting. Interestingly, these boundaries appear largely independent of the specific task, depending instead on the learning rate ratios.

Phase diagram showing the boundaries between representation learning and overfitting based on model parameters.

Conclusion and Future Work

GenEFT provides a powerful new lens for understanding generalization, moving beyond empirical observation toward predictive theory. While currently demonstrated on synthetic graph relations, the framework offers a path toward understanding more complex architectures. Future work will focus on extending the "repon" analogy to high-dimensional, real-world data like natural language and images, and exploring the "Goldilocks Zone" of optimal decoder complexity.

Summary updated

BibTeX
@inproceedings{baek2025geneft,
  author = {David D. Baek and Ziming Liu and Max Tegmark},
  title = {{GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory}},
  booktitle = {Physical},
  year = {2025},
  doi = {10.1103/physreve.111.035307},
}
Figure