Generalization Through Growth: Hidden Dynamics Controls Depth Dependence

Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda

Abstract

This paper introduces a universal, implementation-agnostic framework to explain why deep neural networks consistently outperform shallow ones. By modeling deep learning as a sequence of abstract state transitions, the authors identify four mathematical regimes—EL, EP, PL, and PP—where "Depth Supremacy" (optimal depth $k^* > 1$) is guaranteed. This theory provides a formal justification for the success of modern architectures like Diffusion models and Chain-of-Thought reasoning, showing that optimal depth is driven by the underlying geometry of the task rather than specific hardware or software implementations.

The Challenge of Depth Supremacy

In deep learning theory, a persistent gap exists between mathematical complexity measures and empirical success. Classical theories often suggest that increasing a model's depth should worsen its ability to generalize due to overfitting. However, in practice, deeper models almost always perform better. Most existing explanations for this "depth supremacy" are tied to specific architectures, such as ReLU networks. This research seeks a more fundamental, universal explanation that applies across diverse implementations, from Transformers to Chain-of-Thought prompts.

A Universal State-Transition Framework

The authors recast deep learning through the lens of abstract state-transition semigroups. Instead of analyzing specific weights and neurons, they model a deep network as a sequence of functions (transitions) acting on a metric space. This approach separates the abstract logic of a task from its specific neural implementation.

Deep learning viewed as a sequence of abstract state transitions acting on a metric space.

By analyzing the "covering numbers" (a measure of complexity) of "word balls"—the set of all possible functions created by composing $k$ layers—the authors prove a new version of the bias-variance trade-off. Crucially, they demonstrate that the variance (estimation error) depends on the abstract depth-$k$ transition and is independent of the specific architecture used to realize it.

The Four Regimes of Generalization

The research categorizes model behavior into four distinct regimes based on how the approximation error (bias) decays and how the estimation error (variance) grows with depth $k$:

  • EL (Exponential-Logarithmic): Exponential bias decay and Logarithmic variance growth.
  • EP (Exponential-Polynomial): Exponential bias decay and Polynomial variance growth.
  • PL (Polynomial-Logarithmic): Polynomial bias decay and Logarithmic variance growth.
  • PP (Polynomial-Polynomial): Polynomial bias decay and Polynomial variance growth.
Error dynamics across the four regimes (EL, EP, PL, PP) showing the trade-off between bias and variance as depth increases.

The paper mathematically demonstrates that across almost all these regimes, the optimal depth $k^*$ is greater than 1. This provides a rigorous justification for why shallow learning is rarely the optimal choice for complex tasks.

Implications for Modern AI

The EL regime is identified as the "gold standard" for generalization. This occurs when layers are "contractive," meaning they gradually refine information. This regime perfectly describes the effectiveness of Neural ODEs, Diffusion models, and Chain-of-Thought (CoT) reasoning. These systems use many small, iterative steps to solve a problem, allowing the bias to drop rapidly while keeping complexity (variance) under control. In the EL regime, the optimal depth grows as $O(\log n)$, where $n$ is the number of training samples, achieving a generalization error bound that scales at a rate of $\sqrt{\log(\log(n))/n}$.

Limitations and Future Directions

While the theory provides a powerful existence proof for optimal depth, it currently relies on a "two-stage" learning assumption: finding an abstract minimizer and then embedding it into an architecture. In reality, optimization (like SGD) happens simultaneously. Future work aims to bridge this state-transition view with actual gradient descent dynamics to create a unified theory of deep learning generalization. Furthermore, while the theory proves an optimal depth exists, it does not yet provide a direct algorithm for practitioners to calculate the exact number of layers for a specific real-world dataset.

Summary updated

Abstract

Why and when is deep better than shallow? We answer this question in a framework that is agnostic to network implementation. We formulate a deep model as an abstract state-transition semigroup acting on a general metric space, and separate the implementation (e.g., ReLU nets, transformers, and chain-of-thought) from the abstract state transition. We prove a bias-variance decomposition in which the variance depends only on the abstract depth-$k$ network and not on the implementation (Theorem 1). We further split the bounds into output and hidden parts to tie the depth dependence of the variance to the metric entropy of the state-transition semigroup (Theorem 2). We then investigate implementation-free conditions under which the variance grow polynomially or logarithmically with depth (Section 4). Combining these with exponential or polynomial bias decay identifies four canonical bias-variance trade-off regimes (EL/EP/PL/PP) and produces explicit optimal depths $k^\ast$. Across regimes, $k^\ast>1$ typically holds, giving a rigorous form of depth supremacy. The lowest generalization error bound is achieved under the EL regime (exp-decay bias + log-growth variance), explaining why and when deep is better, especially for iterative or hierarchical concept classes such as neural ODEs, diffusion/score-matching models, and chain-of-thought reasoning.
BibTeX
@misc{sonoda2025generalization,
  author = {Sho Sonoda and Yuka Hashimoto and Isao Ishikawa and Masahiro Ikeda},
  title = {{Generalization Through Growth: Hidden Dynamics Controls Depth Dependence}},
  year = {2025},
  eprint = {2505.15064},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15064},
}
Figure