Measuring the Effective Depth-to-Width Ratio of Real Networks

Two bodies of theory say the same thing in different languages. One sends a network's depth and width to infinity together at a fixed ratio. The other bounds how badly narrow layers destabilize a finite network. The shared message is that a deep network's behavior on any single random initialization is governed by how much small per-layer noise piles up across depth. For a plain ReLU network, this accumulated noise is the log-variance of the gradient norm, and it equals the sum of reciprocal layer widths, $\beta=\sum_l 1/w_l$, which reduces to depth divided by width when all layers have the same width. Architecture designers already use a crude version of this idea: some zero-cost architecture-search methods penalize depth-over-width. But real networks have skip connections, normalization, attention and uneven widths. For these networks nobody knows what the right "depth-to-width ratio" is, and the closed-form $\beta$ is known to be wrong for them.

We propose to stop guessing that ratio and measure it. We will estimate, at initialization, the variance across random weight draws of the log input-output gradient norm. We will then split that variance into a per-block contribution plus a cross-layer correlation term, which gives an empirical "effective width" for every block. The claim is that this measured quantity, which we call $\hat\beta$, is the architecture-general version of $d/n$. We expect it to predict which configurations fail to start training and how widely results vary across seeds. It should do so better than depth-over-width or the analytic $\beta$, and it should be a better trainability constraint inside entropy-based architecture search.

Core Claim

Define $G=\log\|J u\|^2$, where $J$ is the input-output Jacobian of a network at initialization and $u$ is a random unit input direction. Take the variance over independent weight draws and inputs. Write $r_l$ for the log ratio of the Jacobian-vector-product norm after block $l$ to its norm before block $l$. Then $G=\sum_l r_l$, and

$$\hat\beta \equiv \mathrm{Var}(G) = \sum_l \mathrm{Var}(r_l) + \sum_{l\neq m}\mathrm{Cov}(r_l,r_m).$$

The first sum defines per-block effective widths $w^{\rm eff}_l = 1/\mathrm{Var}(r_l)$. The second sum measures cross-layer correlation. The theory makes three sharp predictions about $\hat\beta$ that we can check before relying on it.

First, for ReLU MLPs, Products of Many Large Random Matrices and Gradients in Deep Neural Networks proves that $G$ is approximately Gaussian with variance $\beta=\sum 1/n_i$. It also shows this holds regardless of the weight distribution. So the cross term should vanish and $w^{\rm eff}_l\approx w_l$.

Second, for vanilla ReLU ResNets in the joint limit, The future is log-Gaussian finds outputs modulated by a log-Gaussian scalar. Its variance is inflated by correlations between layers, by a factor of about 250 at $d/n=1$. "Balanced" ResNets, which multiply each ReLU input by a frozen random sign, cut that factor to about 10. So in our decomposition the cross term should dominate for vanilla ResNets and shrink for balanced ones.

Third, for ResNets whose residual branches are scaled by $1/\sqrt{L}$, Width and Depth Limits Commute in Residual Networks shows that pre-activations stay Gaussian and the two limits commute. This matches the finding of How to Start Training that residual coefficients $\eta_\ell$ replace $1/w_l$ in controlling variance. So $\hat\beta$ should stay bounded as depth grows at fixed width.

If $\hat\beta$ reproduces all three regimes, it becomes a single measurable quantity that covers both research directions. We can then carry it to places neither theory reaches: attention blocks, normalization layers, mixed widths, and trained checkpoints.

Why This Is Different From What Exists

The architecture-design papers supply the application. DeepMAD designs CNNs by maximizing a Gaussian-entropy expressiveness score subject to a stage-level bound on $\rho=L/w$. MeRino does the same for sub-100M language models, using an empirically tuned $L/w$ penalty. Both treat the ratio as a heuristic. Neither accounts for skip connections, the varying widths inside a stage, or attention.

Simply substituting Hanin's analytic $\beta$ would be a cosmetic change. By the power-mean argument in Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?, $\beta$ is essentially $L/w$ for constant-width stages. It is also provably the wrong quantity for vanilla or branch-scaled residual networks.

The theory papers motivate the work but also constrain it. The Neural Covariance SDE, Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks and The Shaped Transformer give exact limits only for particular architectures and only at step 0. Hanin's results assume a single input and independent layers. Our contribution is an estimator with a decomposition that is checked against each of these limits and then used as a design and diagnostic tool. We do not claim a new limit theorem.

Research Program

The estimator needs only forward passes and Jacobian-vector products at initialization. For each candidate architecture we draw a few hundred weight seeds and a few input directions, and record the per-block log norm ratios. The cost is a small fraction of a single training step for the full batch of seeds. We will report confidence intervals from bootstrapping over seeds, because variance estimates of heavy-tailed log quantities are noisy. This matters most in the vanilla-ResNet regime.

The program has four parts.

The first is validation against the three theoretical regimes above. We use synthetic MLPs and ResNets on the grids those papers simulated: $n=d=150$ for shaped and unshaped MLPs, and depths and widths from 5 to 5,000 with $10^4$ initializations for residual networks. The target is to reproduce $\mathrm{Var}(G)\approx\sum 1/n_i$ for MLPs and the vanilla-versus-balanced correlation gap.

The second is extension to blocks without theory. We measure $w^{\rm eff}_l$ for Pre-LN, Post-LN and shaped attention blocks and for convolutional stages. We ask how $w^{\rm eff}$ depends on hidden size, head dimension and sequence length. This is an empirical answer to the missing Transformer $\beta$ noted in the file's gap analysis. We will also test whether multi-transformation blocks ($k\ge 2$) behave differently from single-transformation blocks, as the dichotomy in Spectral Condition for μP under Width–Depth Scaling suggests they should.

The third is prediction. The object to predict is trainability, not accuracy in general. That means the fraction of seeds that fail to start training (Hanin and Rolnick's "failure mode 2"), and the spread of early and final loss across seeds.

The fourth is tracking during training. We recompute $\hat\beta$ on checkpoints, where a seed ensemble is replaced by an ensemble of independently trained runs. This asks whether the initial ranking of architectures survives once weights in different layers become correlated. Every theory in the file is silent on this question.

Baselines

The naive reference point is depth divided by width, computed exactly as DeepMAD's stage-level $\rho$. If $\hat\beta$ does not rank architectures better than this ratio, the measurement adds nothing.

The second baseline is the analytic $\beta=\sum_l 1/w_l$ from Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?. Its residual variant is $\sum \eta_\ell$ from How to Start Training. These are the strongest closed-form competitors. They are exact for the architectures they cover, so on plain MLPs we expect a tie, and a tie is the correct outcome there. $\hat\beta$ must win on residual, attention and normalized architectures.

The strongest existing method for architecture design is DeepMAD itself. On ImageNet-1K it reports 82.8% top-1 for DeepMAD-29M, against 82.1% for ConvNeXt-Tiny and 81.3% for Swin-Tiny, and 77.7% for its re-designed ResNet-18 (DeepMAD). For language models, the competitor is MeRino, which reports that MeRino-64M matches OPT-350M accuracy on its zero-shot suite. There is no single aggregate number for this in the research file, so we will re-run MeRino's search as the baseline rather than quote a figure. For the fair comparison, we keep DeepMAD's entropy objective, parameter and FLOP budgets, search algorithm and training recipe. We replace only its $\rho$ constraint with a bound on $\hat\beta$, and we tune the bound's threshold with the same number of trials DeepMAD's $\rho$ threshold receives.

Beating DeepMAD on CIFAR-100 means one of two things at matched budget: higher top-1 accuracy averaged over three seeds with non-overlapping intervals, or equal accuracy with a lower fraction of failed or unstable seeds. We treat ImageNet-1K as a single confirmatory run at the 29M budget. There, "beating" means reaching or exceeding 82.8%.

Two ablations isolate the key ingredient, which is the cross-layer correlation term. The first uses only $\sum_l\mathrm{Var}(r_l)$ and drops the covariances. The second uses a single-seed version of $\hat\beta$, measured across inputs rather than weight draws. This tests whether the quenched, across-seed variance is what matters.

Why not just…?

Why not just use LayerNorm, or $1/\sqrt{L}$ branch scaling and μP, and forget about the ratio? This is the most serious threat, and we take it as a hypothesis to test rather than dismiss. If branch scaling makes $\hat\beta$ bounded, as Width and Depth Limits Commute in Residual Networks implies, then within a well-parameterized family the quantity may carry no ranking signal at all. The evidence that normalization is not a complete fix is limited but real. The Shaped Transformer reports a depth-24 Pre-LN baseline scoring 0.022 on CoLA, against 0.211 for the shaped model. So normalized models at depth can still fail. Our OpenWebText grid will include standard parameterization, the $k\ge2$ μP of Spectral Condition for μP under Width–Depth Scaling, and Pre-LN. If $\hat\beta$ is flat and non-predictive under μP or Pre-LN, we will report that its usefulness is confined to unnormalized, shaped and compact edge models, and narrow the claim accordingly.

Why not just plug in the analytic $\beta=\sum 1/w_l$? It costs nothing and is exact for ReLU MLPs. But the theory already shows it misses what matters for residual networks: the roughly 250-fold correlation inflation in vanilla ResNets (The future is log-Gaussian). It also has no formula for attention. The validation stage and the correlation-term ablation settle directly whether the extra measurement is needed.

Why not just train each candidate for a few hundred steps and use early loss as the proxy? This is a real competitor for predicting final quality, and it may well win on that target. We will compare against it at matched wall-clock compute. Our claim is narrower: $\hat\beta$ predicts seed-to-seed instability, and it does so at a cost cheap enough for a CPU-scale search loop like DeepMAD's under-one-hour budget. A short training run on one seed cannot see variability across seeds at all.

Why not just use conventional architecture search? DeepMAD contrasts its CPU search with 48,000 GPU hours for NASNet-A. The question is therefore which trainability term a cheap search should use, not whether to use search at all.

Proposed Experiments

The first decisive check is the validation grid. On MLPs, measured $\hat\beta$ must match $\sum 1/n_i$ within bootstrap error. On vanilla ResNets, the cross term must account for most of the variance, and this must shrink substantially for balanced ResNets. On $1/\sqrt{L}$-scaled ResNets, $\hat\beta$ must stay bounded as depth grows. Failing any of these would mean the estimator is too noisy to use, and the program stops there.

The second check reuses Hanin's vectorized-MNIST protocol: the number of epochs needed to reach 20% accuracy. Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients? reports that this grows with $\beta$, and that architectures with equal $\beta$ train at equal speed. The research file gives no numeric value for this. We extend the protocol to residual, balanced and normalized MLPs. The test is whether architectures with equal $\hat\beta$ but different $\beta$ train at equal speed.

The third check is a fixed-parameter grid of CNN width profiles on CIFAR-10 and CIFAR-100, including deliberate bottlenecks, plus a GPT-2-style depth-width grid on OpenWebText under the parameterizations above. The metric is Spearman correlation between each proxy ($L/w$, analytic $\beta$, early-loss, $\hat\beta$) and three outcomes: the fraction of five seeds that fail, the across-seed loss spread, and the mean final loss. $\hat\beta$ should clearly win on the first two outcomes, and we make no strong prediction on the third.

The fourth check is the drop-in replacement inside DeepMAD on CIFAR-100, then ImageNet-1K. We also run a small-scale replication of MeRino's search on The Pile with a reduced step budget, evaluated on WikiText-2 perplexity and the HellaSwag, ARC and OpenbookQA zero-shot tasks.

Finally, we recompute $\hat\beta$ at checkpoints throughout the third check. If the architecture ranking at initialization persists through training, a measurement at step 0 is enough to guide design. If the ranking reshuffles, that is the first empirical map of how inverse-width noise evolves under gradient descent, which is exactly the gap every cited theory leaves open.