Theory of very deep networks says that when depth and width grow together, randomness at initialization does not average away. Small per-layer fluctuations of size one-over-width pile up and decide whether a network trains at all. Two literatures describe this pileup in different languages. The first, proportional depth-width scaling, models layer-to-layer propagation as a continuous random process, a stochastic differential equation (SDE) whose "time" runs from 0 to the depth-to-width ratio. Every such derivation assumes all layers have the same width. The second, inverse width scaling, shows that for ReLU networks of any width profile, the spread of gradient sizes across random initializations is set by one number, the summed inverse width $\beta = \sum_l 1/w_l$. Real architectures are rarely constant-width: CNNs double channels stage by stage, and small language models taper. Nobody has said what the continuous-limit theory predicts for them.
We propose that inverse width is the network's natural clock. If each layer's activation shaping (bending the nonlinearity toward the identity by an amount tied to width) is set from that layer's own width, then a network with any width profile should behave at initialization like a constant-width network with the same $\beta$. Its representations should follow one SDE run for "time" $\beta$. If that holds, it splits architecture design in two. $\beta$ is a trainability budget. Within that budget, the shape of the width profile is free to serve representation or efficiency goals. The program derives this equivalence, tests where it breaks (training, attention, normalization), and checks whether trainability curves for very different width profiles fall onto a single curve when plotted against $\beta$.
Core Claim
Consider a shaped network: an MLP, ResNet branch, or attention block in which each layer's nonlinearity has slopes $1 \pm c/\sqrt{n_\ell}$, where $n_\ell$ is that layer's output width. In such a network each layer changes the representation covariance by a drift of order $1/n_\ell$ and a random kick with variance of order $1/n_\ell$. Writing $\tau_\ell = \sum_{k \le \ell} 1/n_k$ for accumulated inverse width, the claim is that as widths grow and the total $\beta$ stays fixed, the covariance $\Sigma$ of the layer outputs over a batch of inputs converges to
$$d\Sigma_\tau = b(\Sigma_\tau)\, d\tau + \sigma(\Sigma_\tau)\, dB_\tau, \qquad \tau \in [0, \beta],$$
where $b$ and $\sigma$ are the drift and diffusion coefficients already derived for constant width in The Neural Covariance SDE, and $B$ is a matrix Brownian motion. The new content is the time change. Width profile enters only through $\tau$, and the constant-width horizon $d/n$ is replaced by $\beta$. Three corollaries follow.
First, any two shaped profiles with equal $\beta$ have the same limiting law at initialization, and their gradient log-norm distributions match as well.
Second, the clock must count each layer's own width. Shaping all layers with the network's average width breaks the equivalence. This is the ingredient our ablation isolates.
Third, for unshaped ReLU the same bookkeeping should recover the log-normal law $\ln\|M\vec u\|^2 \approx \mathcal N(-\beta/2, \beta)$ of Products of Many Large Random Matrices and Gradients in Deep Neural Networks. That result is our consistency check.
How This Relates to Prior Work
Two papers supply the method and motivate the gap. The Neural Covariance SDE treats propagation as a Markov chain converging to an SDE, and Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks shows that shaped MLPs and suitably scaled ResNets share a limit. Both fix $n_\ell \equiv n$. We reuse their per-layer expansions with layer-dependent width, which is exactly the setting they leave out.
Hanin's line of work supplies the invariant. Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients? bounds Jacobian moments by $\exp(C\beta)$ for arbitrary widths, and How to Start Training separates mean drift, which initialization fixes, from variance blow-up, which only architecture fixes. These results give moment bounds for single input vectors in unshaped ReLU nets. They do not give a process-level description of multi-input covariance, and they do not cover shaping or attention.
Three papers constrain the design. The future is log-Gaussian shows vanilla ResNets at depth equal to width have an inter-layer correlation variance factor of about 250, reduced to about 10 by random sign balancing. So our ResNet experiments must include the balanced variant, or the clock will be confounded with hypoactivation (fewer than half of the ReLUs firing). The Shaped Transformer supplies attention shaping (centered softmax, an identity bias, temperature tied to width), but only for uniform width. Our extension makes the temperature and slope local to each block.
DeepMAD and MeRino use a depth-over-width ratio as a zero-cost trainability constraint during architecture search. We test whether $\beta$ under local shaping is a sharper version of that heuristic. Finally, there is the power-mean result that constant width minimizes $\beta$ at a fixed parameter count. This does not contradict us. It says non-uniform profiles pay a $\beta$ price, and our claim is that this price is the only trainability cost they pay.
Research Program
The theory step derives the time-changed SDE for shaped MLPs with stage-wise and smoothly varying widths. It has to handle the rectangular weight matrices where width changes, and it has to determine whether fan-in or fan-out width sets the per-layer noise. Our derivation predicts output width, since each layer's covariance is an average over its own neurons. We will test this directly by comparing a fan-in clock with a fan-out clock. We then extend the derivation to residual branches and to shaped attention blocks with local temperatures.
The simulation step compares Monte Carlo covariance and gradient-norm distributions across many seeds against numerical solutions of the SDE. The empirical step trains families of matched-$\beta$ and matched-parameter architectures and asks whether trainability and final quality are functions of $\beta$ alone. That is the question of [Gap 4]: does the initialization theory survive feature learning?
Baselines
The naive reference point is the standard unshaped network with He initialization and the same width profile. Hanin's results predict it degrades with $\beta$, but with no matched-time SDE behind it. It is run with identical data, optimizer, and learning-rate sweep. Beating it means trainability improves at large $\beta$, which is necessary but not sufficient.
The strongest existing method for deep, normalization-free training is the uniform-width Shaped Transformer of The Shaped Transformer, which reports a CoLA Matthews correlation of 0.211 at depth 24, against 0.022 for a Pre-LN Transformer. For non-uniform profiles we run it in two ways: with a single global shaping constant computed from mean width, and in its original constant-width form at equal parameters. All variants get the same tuning budget, a grid over learning rate and the shaping constant $c$. Beating it means a tapered or staged profile with local shaping matches or exceeds 0.211 on CoLA at depth 24 or deeper, with comparable parameters. It also means trainability curves across profiles collapse onto one curve in $\beta$, where the global-shaping version's curves do not.
The normalization baseline is Pre-LN on every profile, at equal compute and tuning. Beating it means comparable quality without normalization, plus a predictive theory it lacks.
The key ablation is local versus global shaping at identical profile and $\beta$. If global shaping performs the same, the second corollary is false and the proposal collapses to "use any shaping." The fan-in versus fan-out clock ablation tests the derivation's sharpest prediction.
For architecture search we compare against DeepMAD's $L/w$ constraint as a predictor of trainability. There is no reported result on Lacuna for numerical CIFAR-100 accuracies in DeepMAD; the paper reports only a figure. So we reproduce its constraint ourselves rather than quoting a number. The DeepMAD ImageNet-1K figure of 82.8% top-1 at 29M parameters is context, not a target, because we do not plan ImageNet-scale search.
Why not just…?
Why not just use LayerNorm and stop worrying about width profiles? Normalization is the real practical threat at moderate depth, which is why Pre-LN is a baseline on every profile. The evidence that it falls short at depth comes from The Shaped Transformer: query and value gradients vanish in deep Pre-LN and Post-LN models, and CoLA drops to 0.022 at depth 24. Normalization also gives no prediction of where training will fail. If Pre-LN matches local shaping at every depth we test, we will report that the clock is a descriptive theory rather than a design tool.
Why not just use constant width, since it minimizes $\beta$? At a fixed parameter count this is correct for stability, per Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?. But staged CNNs and tapered models exist for other reasons, such as compute per spatial resolution and inference latency, as in DeepMAD and MeRino. Our contribution is to price their stability cost exactly. Matched-parameter comparisons will show whether non-uniform profiles gain anything once that cost is paid. If they never do, that is a clean negative result, and we will state it.
Why not just use the depth-over-width heuristic from DeepMAD or MeRino? For constant width it coincides with $\beta$, so it can only differ on non-uniform profiles. That is exactly where we test it. If $L/\bar w$, with $\bar w$ the mean width, predicts epochs-to-train as well as $\beta$ across profiles, the theory adds little in practice. The profile-collapse experiment settles this directly.
Why not just balance signs as in Balanced ResNets? Sign balancing removes hypoactivation, but per The future is log-Gaussian a variance factor of about 10 remains at depth equal to width. That residual is the width-driven diffusion our clock describes. We therefore treat balancing as complementary and include balanced ResNets in every ResNet arm.
Proposed Experiments
The initialization study builds MLPs, balanced ResNets, and shaped Transformers with five width profiles: constant, increasing, decreasing, hourglass, and channel-doubling stages. Each profile is constructed at three values of $\beta$ spanning 0.25 to 2. We measure, over at least 1,000 seeds, the Kolmogorov-Smirnov distance between empirical input-pair correlations and SDE solutions in $\tau$. We also measure the distribution of gradient log-norms against Hanin's $\mathcal N(-\beta/2,\beta)$ prediction in the unshaped case. The decisive check is that matched-$\beta$ profiles are statistically indistinguishable under local shaping and distinguishable under global shaping.
The training study first uses Hanin's vectorized MNIST benchmark of epochs to reach 20% accuracy. Earlier papers report a near-perfect correlation with $\beta$ but plot it without tabulated numbers, so we regenerate it for all profiles. We then move to CIFAR-10 and CIFAR-100 with staged CNNs, GLUE CoLA with shaped Transformers at depths 24 to 96, and sub-100M language models on The Pile at MeRino's scale, evaluating perplexity on WikiText-2. Across all profiles we plot final loss and trainability against $\beta$ and against $L/\bar w$, and quantify how tightly the curves collapse. Tracking the covariance distribution over the course of training tells us when feature learning breaks the $\beta$-equivalence. The proposal succeeds if collapse in $\beta$ holds through training at least in the regime where profiles train comparably, and if the local-versus-global ablation shows a measurable gap. It fails if mean-width heuristics predict equally well.
References
- The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
- Products of Many Large Random Matrices and Gradients in Deep Neural Networks
- Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
- Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
- How to Start Training: The Effect of Initialization and Architecture
- The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization
- The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
- DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network
- MeRino: Entropy-Driven Design for Generative Language Models on IoT Devices