Abstract
Following the recent work on capacity allocation, we formulate the conjecture that the shattering problem in deep neural networks can only be avoided if the capacity propagation through layers has a non-degenerate continuous limit when the number of layers tends to infinity. This allows us to study a number of commonly used architectures and determine which scaling relations should be enforced in practice as the number of layers grows large. In particular, we recover the conditions of Xavier initialization in the multi-channel case, and we find that weights and biases should be scaled down as the inverse square root of the number of layers for deep residual networks and as the inverse square root of the desired memory length for recurrent networks.
Introduction: The Shattering Problem
As neural networks grow deeper, they often encounter the shattering problem: the mapping from input to output becomes so complex and "noisy" that gradients used for training look like white noise, making optimization nearly impossible. While practitioners have found empirical "tricks" to fix this—such as residual connections (ResNets) and specific initialization schemes (Xavier/He)—we lack a unified theoretical explanation for why these specific scales work. This research seeks to move beyond trial-and-error by using "capacity allocation analysis" to determine the exact mathematical conditions required for a network to maintain a stable, non-degenerate flow of information from the first layer to the last.
Methods: Capacity Allocation and the Continuous Limit
The core methodology involves treating a network's parameters as a finite resource distributed across the input space, termed capacity allocation.
- Continuous Limit Modeling: The paper treats a deep network as a continuous system where the number of layers ($L$) tends to infinity.
- PDE Framework: By assuming a "pseudo-random" limit (where activations are highly irregular), the author shows that capacity propagation follows a discrete Markov equation. In the deep limit, this becomes a continuous Kolmogorov forward equation (a type of diffusion PDE).
- Scaling Derivation: The author analyzes various architectures to find the "Goldilocks" scaling relations. If weights are too large, the system "shatters" (explodes); if they are too small, capacity fails to propagate (vanishes).
Theoretical Framework and Scaling Laws
The Neural Network Scaling Conjecture argues that deep networks only avoid shattering if their capacity to model data propagates through layers like a well-behaved physical diffusion process.
- The $1/\sqrt{L}$ Rule: For residual networks, the residual weights must scale as the inverse square root of the number of layers ($1/\sqrt{L}$). This ensures the network behaves like a diffusion process, where the "receptive field" grows predictably as $\sqrt{L}$.
- Unifying Xavier Initialization: The framework recovers the well-known Xavier initialization ($1/\sqrt{C}$, where $C$ is the number of channels) as a specific case of capacity conservation in multi-channel networks.
- Capacity Leaks: The paper introduces the concept of a "capacity leak" when providing inputs to intermediate layers (common in GANs or conditioned models). It suggests that these intermediate inputs must also be scaled by $1/\sqrt{L}$ to prevent the original input's influence from vanishing.
- RNN Memory: For Recurrent Neural Networks, the weights should scale as $1/\sqrt{N}$, where $N$ is the desired memory length, suggesting a theoretical path toward achieving longer-term dependencies.
Summary of Derived Scaling Relations
The paper provides a "cheat sheet" of scaling laws required to avoid shattering across different architectures:
- Vanilla ResNets: Weights must scale as $1/\sqrt{L}$.
- Multi-channel Networks: Recovers the scaling $1/\sqrt{C}$, aligning with standard initialization practices.
- Biases: Biases should not be constant; they should also be scaled down by $1/\sqrt{L}$ in very deep networks to maintain non-degenerate propagation.
- Multi-dimensional Inputs: For $d$-dimensional inputs (like images) with convolutional filters of size $r$, the scaling law adjusts to $1/\sqrt{r^d L}$.
Discussion and Future Work
While this framework provides a powerful "physics-based" rule for designing deep architectures, it has limitations. The derivations rely on the "pseudo-random limit," assuming very irregular activation functions. Real-world activations like ReLU or Sigmoid are smoother, and the results need to be generalized to these types. Furthermore, the paper is purely theoretical; empirical validation on standard datasets is needed to confirm if these specific scalings, particularly for biases and intermediate inputs, outperform existing practices. Finally, the RNN analysis assumes "local" state transformations, which differs from standard LSTM implementations and warrants further investigation.
Summary updated