Abstract
This research establishes a theoretical bridge between statistical physics and deep learning to resolve the "curse of dimensionality." By identifying locality and compositionality as the fundamental structural properties of natural data, we demonstrate how specific architectures—such as Convolutional Neural Networks (CNNs) and Diffusion Models—exploit these properties to learn efficiently. Through the introduction of the Random Hierarchy Model (RHM) and the analysis of weight space linearization, this work quantifies how models adapt to spatial scales, undergo generative phase transitions, and organize tasks into modular, editable directions in weight space.
Introduction
In classical learning theory, the "curse of dimensionality" suggests that the volume of data required to learn a task grows exponentially with the number of input dimensions. For high-resolution images or complex linguistic structures, this would imply a requirement for astronomical amounts of data. However, deep learning models routinely achieve high performance with relatively sparse datasets. This paper addresses this paradox by investigating the latent structure of data. We argue that natural data is not arbitrary; it is governed by locality (where proximal features are most relevant) and compositionality (where complex objects are built from simpler, hierarchical parts).
Methodology
To analyze these phenomena, we employ a multi-disciplinary toolkit:
- Statistical Mechanics: We utilize the replica method and mean-field theory to analyze learning dynamics and generalization error in high-dimensional regimes.
- Infinite-Width Limits: By analyzing models in the "lazy" regime—where they behave like Neural Tangent Kernels (NTK)—we make complex neural dynamics analytically tractable.
- The Random Hierarchy Model (RHM): We propose a novel synthetic data framework based on probabilistic context-free grammars. This model simulates hierarchical data (like language or images) where high-level symbols generate tuples of lower-level symbols, allowing for controlled experiments on data structure.
- Weight Space Linearization: We use first-order Taylor expansions to study "task vectors," defined as the difference in weights between a pre-trained model and a fine-tuned one, to understand how knowledge is organized in the parameter space.
Theoretical Framework and Key Insights
The Primacy of Locality
Our analysis of CNNs reveals that the local receptive field—rather than weight sharing—is the primary mechanism for overcoming the curse of dimensionality. We derive a learning curve exponent $\beta = \alpha_t/s$, where $s$ represents the filter size. This shows that generalization improves at a rate determined by the local filter size rather than the total input dimension $d$, allowing for polynomial rather than exponential scaling.
Generative Phase Transitions in Diffusion
In diffusion models, we identify a critical noise threshold that governs the generative process. Below this threshold, the model preserves high-level semantic features (e.g., the specific class of an object). Above this threshold, a phase transition occurs: the specific class identity is lost, but the model retains the ability to "creatively" recombine low-level features to compose entirely new objects. This suggests that diffusion models learn a hierarchical "grammar" of data.
Generalization vs. Memorization
We find that memorization in diffusion models is a dynamical process. Models tend to learn the underlying distribution (generalization) before they begin to encode specific training examples (memorization). This temporal separation provides a principled basis for early stopping as a defense against copyright and privacy infringements.
Weight Disentanglement and Task Arithmetic
Pre-training serves to organize a model’s weight space into functionally independent regions. This "disentanglement" allows for Task Arithmetic, where adding or subtracting weight vectors corresponds to adding or removing specific model capabilities. By fine-tuning models in their linearized "tangent space," we can achieve superior model editing results.
Empirical Results
- Sample Complexity: We prove that learning hierarchical grammars scales polynomially with data dimension, providing a mathematical explanation for the efficiency of diffusion models.
- Model Editing Performance: Using a suite of eight tasks (including Cars, DTD textures, and EuroSAT), we demonstrated that linearized fine-tuning improves multi-task accuracy by up to 5.8 points and reduces unwanted task "forgetting" by up to 13.1 points compared to standard methods.
- Validation on Real Data: Empirical tests on Masked Language Models (MDLM) and Denoising Diffusion Probabilistic Models (DDPM) confirmed the existence of a "peaking correlation length" at the predicted phase transition, validating the RHM as a faithful proxy for real-world data patterns.
Discussion and Limitations
While this work provides a robust framework for understanding the "lazy" regime of deep learning, it does not fully capture "feature learning," where internal representations change significantly. Future research must address the emergence of features from scratch. Additionally, while the RHM is a powerful tool, it currently relies on a fixed tree structure; extending this to context-sensitive, irregular topologies will be essential for a complete theory of natural language processing. Finally, identifying which specific pre-training objectives (e.g., contrastive vs. generative) most effectively drive weight disentanglement remains an open question.
Summary updated