Extending Deep Learning Theory to Complex Architectures

334 concepts305 papers2 proposals

Deep learning theory faces a persistent uncomfortable truth: the mathematical frameworks used to explain why neural networks work are often trapped in the land of Multi-Layer Perceptrons (MLPs).

When theorists want to prove that a network will generalize to unseen data, optimize efficiently, or learn meaningful representations, they typically assume a simple feedforward structure. In this idealized setting, signals propagate through a clean sequence of independent matrix multiplications and coordinate-wise activations.

But modern deep learning does not use simple MLPs. It relies on Transformers, Residual Networks (ResNets), and Convolutional Neural Networks (CNNs). These architectures introduce complex structural mechanisms—such as data-dependent attention routing, skip connections, layer normalization, and localized weight sharing. This divergence creates an architectural generalization gap within theory itself: our most rigorous mathematical guarantees frequently break down when applied to the models actually used in practice.

The Tractability Baseline: Infinite Width and Feature Learning

To make deep networks mathematically manageable, researchers often study them in the "infinite-width limit." Under standard parameterizations, as the layers of a network become infinitely wide, its behavior simplifies.

In a foundational result, Greg Yang proved in Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes that almost any architecture converges to a Gaussian Process (GP) at initialization. This unified framework extended earlier deep MLP proofs, such as those derived by Alexander G. D. G. Matthews in Gaussian Process Behaviour in Wide Deep Neural Networks, to modern structures like RNNs and Transformers.

However, standard infinite-width limits (like the Neural Tangent Kernel, or NTK) suffer from a major conceptual flaw: they freeze the network's internal representations during training. To capture true "feature learning"—the process that allows deep networks to excel at transfer tasks—Yang et al. introduced the Maximal Update Parametrization ($\mu P$) in Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks. This framework provides a recipe for scaling networks while preserving feature learning dynamics.

Standard NTK Limit:   Width → ∞  |  Representations Freeze (No Feature Learning)
Maximal Update (μP):  Width → ∞  |  Representations Evolve (True Feature Learning)

The friction point arises when trying to extend these limits beyond basic MLPs. For instance, Littwin et al. demonstrated in Adaptive Optimization in the $\infty$-Width Limit that while adaptive optimizers like Adam can be rigorously modeled at infinite width for MLPs, extending these proofs to Transformers remains an open challenge.

Similarly, Yang et al. noted in A theory of representation learning gives a deep generalisation of kernel methods that while representation learning limits exist for fully connected and convolutional layers, deriving tractable kernel equivalents for Transformers or recurrent networks is exceptionally difficult due to their specific connectivity patterns.

Why the Mathematics Breaks Down

Why do elegant mathematical proofs fail to cross over to complex architectures? The answer lies in how structural components violate the core assumptions of statistical learning theory.

1. Dynamic Routing vs. Static Propagation

In an MLP, the path a signal takes through the network is static; it is determined solely by fixed weight matrices and coordinate-wise activations. In a Transformer, the attention mechanism introduces data-dependent routing. The network computes a dot-product between queries and keys to create an attention matrix, which then scales the values.

This means activations are multiplied by other activations, introducing a severe non-linearity that standard signal propagation theories—which assume weights and inputs are independent—cannot easily parse. As Boris Hanin showed in Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?, gradient stability in deep ReLU networks can be governed by a single architectural parameter based on layer widths. However, this clean formulation breaks down when attention mechanisms or residual connections dynamically alter the effective depth and width of the computation graph.

2. The Breakdown of Generalization Bounds

Statistical generalization bounds often rely on measuring the capacity of a function class (e.g., via Rademacher complexity or PAC-Bayes frameworks). Dziugaite et al. demonstrated in Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data that it is possible to compute meaningful, nonvacuous bounds for overparameterized models.

Yet, these calculations are heavily tailored to fully connected layers. When Pour et al. explored generalization bounds based on additive noise in Benefits of Additive Noise in Composing Classes with Bounded Capacity, they explicitly noted that the applicability and tightness of these metrics for high-parameter, structured architectures like Transformers or ResNets remain an open question. The intricate dependencies introduced by normalization layers and residual blocks prevent standard norm-based regularizers from providing tight bounds.

[Input] → [LayerNorm] → [Attention] → [Residual Add] → [Output]
               ↑                            │
               └────────────────────────────┘
         (Breaks standard i.i.d. layer assumptions)

3. Initial Alignment and Inductive Biases

For a network to learn via gradient descent, it must be capable of establishing an early correlation with the target function. Emmanuel Abbe formalized this in An Initial Alignment between Neural Network and Target is Needed for Gradient Descent to Learn, proving that without a non-negligible "Initial Alignment" (INAL), even massive architectures fail to learn target functions in polynomial time.

While this necessity is proven for fully connected networks, complex architectures bypass or alter this constraint through structural inductive biases. A CNN's translation invariance or a Transformer's permutation equivariance acts as a geometric constraint on the search space, effectively forcing an initial alignment that the math for unconstrained MLPs cannot predict.

Bridging the Gap

Resolving the architectural generalization gap is not merely a matter of mathematical bookkeeping; it is essential for making deep learning predictable. Without theories that encompass complex architectures, practical advancements in scaling, optimization, and safety will remain driven by empirical trial-and-error.

Current research is actively working to translate these foundational insights—moving step-by-step from MLPs to CNNs, and ultimately toward fully capturing the non-linear, dynamic mechanics of modern Transformers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 288 papers
  2. Version 1 · 2026-10-04 · 249 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_architectural_scope_limitation_architect_8872a861,
  title = {architectural scope limitation (architectural generalization gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-scope-limitation-architectural-generalization-gap/rdr_8872a8619a544ea689113f31721ae840},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_architectural_scope_limitation_architect_8872a861_v2,
  title = {architectural scope limitation (architectural generalization gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-scope-limitation-architectural-generalization-gap/rdr_8872a8619a544ea689113f31721ae840/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake