The Physics of Data and Tasks: Theories of Locality and Compositionality in Deep Learning

Alessandro FaveroarXiv 2025 · DOI · arXiv

Abstract

This research establishes a theoretical bridge between statistical physics and deep learning to resolve the "curse of dimensionality." By identifying locality and compositionality as the fundamental structural properties of natural data, we demonstrate how specific architectures—such as Convolutional Neural Networks (CNNs) and Diffusion Models—exploit these properties to learn efficiently. Through the introduction of the Random Hierarchy Model (RHM) and the analysis of weight space linearization, this work quantifies how models adapt to spatial scales, undergo generative phase transitions, and organize tasks into modular, editable directions in weight space.

Introduction

In classical learning theory, the "curse of dimensionality" suggests that the volume of data required to learn a task grows exponentially with the number of input dimensions. For high-resolution images or complex linguistic structures, this would imply a requirement for astronomical amounts of data. However, deep learning models routinely achieve high performance with relatively sparse datasets. This paper addresses this paradox by investigating the latent structure of data. We argue that natural data is not arbitrary; it is governed by locality (where proximal features are most relevant) and compositionality (where complex objects are built from simpler, hierarchical parts).

Methodology

To analyze these phenomena, we employ a multi-disciplinary toolkit:

  • Statistical Mechanics: We utilize the replica method and mean-field theory to analyze learning dynamics and generalization error in high-dimensional regimes.
  • Infinite-Width Limits: By analyzing models in the "lazy" regime—where they behave like Neural Tangent Kernels (NTK)—we make complex neural dynamics analytically tractable.
  • The Random Hierarchy Model (RHM): We propose a novel synthetic data framework based on probabilistic context-free grammars. This model simulates hierarchical data (like language or images) where high-level symbols generate tuples of lower-level symbols, allowing for controlled experiments on data structure.
  • Weight Space Linearization: We use first-order Taylor expansions to study "task vectors," defined as the difference in weights between a pre-trained model and a fine-tuned one, to understand how knowledge is organized in the parameter space.

Theoretical Framework and Key Insights

The Primacy of Locality

Our analysis of CNNs reveals that the local receptive field—rather than weight sharing—is the primary mechanism for overcoming the curse of dimensionality. We derive a learning curve exponent $\beta = \alpha_t/s$, where $s$ represents the filter size. This shows that generalization improves at a rate determined by the local filter size rather than the total input dimension $d$, allowing for polynomial rather than exponential scaling.

Generative Phase Transitions in Diffusion

In diffusion models, we identify a critical noise threshold that governs the generative process. Below this threshold, the model preserves high-level semantic features (e.g., the specific class of an object). Above this threshold, a phase transition occurs: the specific class identity is lost, but the model retains the ability to "creatively" recombine low-level features to compose entirely new objects. This suggests that diffusion models learn a hierarchical "grammar" of data.

Generalization vs. Memorization

We find that memorization in diffusion models is a dynamical process. Models tend to learn the underlying distribution (generalization) before they begin to encode specific training examples (memorization). This temporal separation provides a principled basis for early stopping as a defense against copyright and privacy infringements.

Weight Disentanglement and Task Arithmetic

Pre-training serves to organize a model’s weight space into functionally independent regions. This "disentanglement" allows for Task Arithmetic, where adding or subtracting weight vectors corresponds to adding or removing specific model capabilities. By fine-tuning models in their linearized "tangent space," we can achieve superior model editing results.

Empirical Results

  • Sample Complexity: We prove that learning hierarchical grammars scales polynomially with data dimension, providing a mathematical explanation for the efficiency of diffusion models.
  • Model Editing Performance: Using a suite of eight tasks (including Cars, DTD textures, and EuroSAT), we demonstrated that linearized fine-tuning improves multi-task accuracy by up to 5.8 points and reduces unwanted task "forgetting" by up to 13.1 points compared to standard methods.
  • Validation on Real Data: Empirical tests on Masked Language Models (MDLM) and Denoising Diffusion Probabilistic Models (DDPM) confirmed the existence of a "peaking correlation length" at the predicted phase transition, validating the RHM as a faithful proxy for real-world data patterns.

Discussion and Limitations

While this work provides a robust framework for understanding the "lazy" regime of deep learning, it does not fully capture "feature learning," where internal representations change significantly. Future research must address the emergence of features from scratch. Additionally, while the RHM is a powerful tool, it currently relies on a fixed tree structure; extending this to context-sensitive, irregular topologies will be essential for a complete theory of natural language processing. Finally, identifying which specific pre-training objectives (e.g., contrastive vs. generative) most effectively drive weight disentanglement remains an open question.

Summary updated

Abstract

Deep neural networks have achieved remarkable success, yet our understanding of how they learn remains limited. These models can learn high-dimensional tasks, which is generally statistically intractable due to the curse of dimensionality. This apparent paradox suggests that learnable data must have an underlying latent structure. What is the nature of this structure? How do neural networks encode and exploit it, and how does it quantitatively impact performance - for instance, how does generalization improve with the number of training examples? This thesis addresses these questions by studying the roles of locality and compositionality in data, tasks, and deep learning representations.
BibTeX
@misc{favero2025physics,
  author = {Alessandro Favero},
  title = {{The Physics of Data and Tasks: Theories of Locality and Compositionality in Deep Learning}},
  year = {2025},
  eprint = {2510.06106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06106},
}
Figure