A randomly initialized deep network does not have one gradient size. It has a random one that changes with the seed. In narrow, deep ReLU networks with independent Gaussian weights, the logarithm of the gradient norm behaves like a sum of small independent shocks, one per layer, each of size about one over that layer's width. The gradient on any single run is therefore log-normal. Its spread is governed by $\beta=\sum_l 1/w_l$, the sum of reciprocal layer widths, and once $\beta$ is large most runs start with near-zero gradients. This is why such networks need width to grow linearly with depth. Orthogonal initialization breaks this rule in linear networks, where the width needed for fast training becomes independent of depth. Nobody has said what replaces $\beta$ once the nonlinearity is restored and layer widths are allowed to differ, which is the case for every real architecture.
We propose that, under orthogonal initialization, $\beta$ should be replaced by an effective fluctuation budget. In this budget, layers that keep their width or grow it add only a nonlinearity-dependent term, and layers that shrink the width add a term set by how much they shrink. The consequence is concrete and checkable. Two architectures with identical $\beta$ should be equally trainable under Gaussian initialization but very differently trainable under orthogonal initialization, depending on where their narrowings sit. If this holds, the linear width-depth law becomes a law about contractions and activation noise, not about width per se. That would change how fixed neuron budgets should be allocated.
Core Claim
Hanin's analysis in Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients? shows that Jacobian fluctuations grow like $\exp(C\beta)$ for ReLU networks with independent weights, and it explicitly excludes orthogonal matrices. How to Start Training then proves that no change to the variance of independent weights fixes this "exponential variance" failure. That proof leaves structured, non-independent weights untouched. Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks shows that orthogonality removes the depth dependence entirely, but only for linear, constant-width networks.
We decompose the per-layer shock into two sources. The first is the weight matrix, which under Gaussian initialization rescales the signal norm by a chi-square-like factor. The second is the activation, which randomly masks or bends coordinates. An orthogonal square matrix contributes no norm fluctuation at all. A random orthogonal projection from $w_l$ down to $w_{l+1}<w_l$ dimensions retains a Beta-distributed fraction of the squared norm, and the variance of its log is approximately $2(1/w_{l+1}-1/w_l)$. An isometric embedding into more dimensions contributes nothing. This motivates the prediction
$$\beta_{\text{eff}}=\sum_l \frac{\kappa_\phi}{w_l}+2\sum_l \max\!\Big(0,\ \frac{1}{w_{l+1}}-\frac{1}{w_l}\Big),$$
where $\kappa_\phi$ is a constant fixed by the activation's derivative dispersion and measured once per activation. For a linear network, $\kappa_\phi=0$, which recovers Hu et al.'s depth-independence. For tanh run near the origin, $\kappa_\phi$ should be small. For ReLU it should be positive but below the Gaussian-initialized value. For the looks-linear parameter-sharing construction of Initialization of ReLUs for Dynamical Isometry, it should be near zero at step zero. The claim is that the variance of the quenched log-gradient norm (its value on a single run) tracks $\beta_{\text{eff}}$. The claim is also that trainability within a fixed budget follows $\beta_{\text{eff}}$ rather than $\beta$, so the required width scales as $W\propto \kappa_\phi D$ and not $W\propto D$.
The projection term is derived. The $\kappa_\phi/w_l$ form is a hypothesis to be tested, including whether $\kappa_\phi$ is truly constant across depth or drifts as representations evolve. The claim is consistent with On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization. That paper finds orthogonal and Gaussian initializations identical at infinite width, which is exactly where every term above vanishes. The proposal describes the $O(1/w)$ regime that paper leaves unparameterized.
Research Program
The first component is analytical. We will make the projection term rigorous for deep linear networks with arbitrary width profiles, extending Hu et al. beyond uniform width. We will then derive $\kappa_\phi$ for ReLU and tanh under Haar-orthogonal weights, treating the pre-activation coordinates as nearly exchangeable after an orthogonal rotation. Following Vanishing Curvature in Randomly Initialized Deep ReLU Networks, we will work with medians and quenched quantities rather than expectations, because expectations hide exactly this phenomenon.
The second component is measurement. For each combination of activation, initialization, width profile and depth, we sample on the order of a thousand seeds. For each seed we record the log input-output Jacobian norm and the log loss-gradient norm. As a curvature proxy we also record the Gradient Quotient of MetaInit, which measures how much the gradient changes after one step. The activations are linear, tanh and ReLU. The initializations are Gaussian He, Haar orthogonal (delta-orthogonal for convolutions), and looks-linear. We will fit $\kappa_\phi$ on constant-width networks only, then predict variance on held-out non-uniform profiles with no further fitting.
The third component turns the budget into a design rule. Under a fixed neuron count, Hanin's power-mean argument says uniform width is optimal for Gaussian initialization. Our formula says this is no longer true under orthogonal initialization. Under orthogonal weights, expansions are free and a narrowing is paid for once, by its contraction. We will compute the $\beta_{\text{eff}}$-minimizing profile under bottleneck constraints, such as a required narrow layer, and test whether it beats the uniform-width answer.
Baselines
The naive reference point is Hanin's $\beta$ with Gaussian He initialization, used as a predictor of seed-level gradient spread and of time to reach 20% accuracy on vectorized MNIST. This is the metric used in Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?. That paper reports the qualitative finding that training speed tracks $\beta$ for both uniform and alternating widths. No reported numeric value was found on Lacuna for this benchmark, so we will reproduce it ourselves before comparing.
The strongest competing predictor is $\beta$ with a separately fitted constant for each initialization. This is the most natural way to "patch" Hanin's result, and it is also the key ablation. It keeps everything in our proposal except the asymmetric treatment of contraction versus expansion. We beat it only if $\beta_{\text{eff}}$ explains substantially more of the seed-level log-variance and the trainability ordering on held-out non-uniform profiles. The decisive set is architectures matched in $\beta$ but differing in where they contract.
A second architectural predictor is DeepMAD's depth-to-width ratio $\rho=L/w$ from DeepMAD. It is the trainability constraint actually used in practice, but it ignores initialization altogether.
On the initialization side, we include MetaInit as the strongest learned scale-tuning method. The reported result we cite is 3.7% CIFAR-10 test error for a 28-layer plain CNN without normalization, versus 6.0% with batch normalization (MetaInit). For residual settings we cite RISOTTO's 60.63 ± 0.28% accuracy on CIFAR-100 for a ResNet-50 without batch normalization (Dynamical Isometry for Residual Networks). We compare against these only as reference points, not as predictors.
All predictors are evaluated on the same seed pools. All trained models share the same data, optimizer, learning-rate grid and epoch budget. The only differences are the width profile and the initialization.
Why not just…?
Why not just measure the Jacobian spectrum with free probability, as in dynamical isometry? Resurrecting the sigmoid predicts the singular value distribution in the limit of large width. That describes the average spectrum, not how much it fluctuates from seed to seed at finite width, and it is silent on unequal widths. Our claim concerns the finite-width, single-run spread. If the free-probability spectral width alone ranked our held-out profiles as well as $\beta_{\text{eff}}$, the claim would be redundant. We will compute it and report that comparison.
Why not just measure gradient variance empirically at initialization and skip the theory? This is a real threat as a diagnostic. A few dozen seeds may predict trainability as well as any formula. We therefore treat empirical seed variance as an oracle predictor and ask whether a closed-form $\beta_{\text{eff}}$, computed from the architecture alone, matches it. The value of the formula is the design rule it yields: optimal width profiles can be read off without training or sampling anything. That value is tested directly in the budget-allocation experiment.
Why not just use MetaInit, Adam, or normalization so the issue disappears? These change the remedy, not the diagnosis. How to Start Training proves that scale changes to independent weights cannot fix exponential variance. MetaInit tunes only scales, so on profiles with high $\beta$ but low $\beta_{\text{eff}}$ we expect orthogonal initialization to succeed where MetaInit on Gaussian weights does not. Adam rescues deep ReLU MLPs in Vanishing Curvature, so we report results for both SGD and Adam. If Adam erases every difference between profiles, the practical stakes are confined to SGD and low-precision regimes, and we will say so.
Proposed Experiments
The first experiment uses synthetic random unit-vector inputs, as in How to Start Training. We sweep depth up to several hundred, width from 16 to 1024, and width profiles that are uniform, alternating, single-bottleneck and pyramidal. The success metric is the fraction of variance in the seed-level log-gradient norm explained by $\beta_{\text{eff}}$ versus the per-initialization-$\beta$ baseline, on profiles not used to fit $\kappa_\phi$.
The decisive check is a family of profiles matched in $\beta$. They have the same sum of reciprocal widths but contract either once sharply or many times gently, the latter being alternating widths. Under Gaussian initialization these should be indistinguishable. Under orthogonal initialization, $\beta_{\text{eff}}$ predicts the alternating profile pays a contraction cost at every second layer while the single bottleneck pays once. If the observed ordering does not flip between initializations, the asymmetry claim is false.
The second experiment trains MLPs on vectorized MNIST and Fashion-MNIST and plain CNNs on CIFAR-10. We measure the fraction of seeds reaching 20% accuracy within a fixed epoch budget and the median time to reach it, tracing a trainability phase boundary in the width-depth plane. Hu et al. report a diagonal failure boundary for Gaussian initialization and a vertical one for orthogonal initialization in linear networks. We predict the boundary for orthogonal ReLU to be diagonal with a slope set by $\kappa_\phi$, and the boundary for looks-linear ReLU to be nearly vertical.
The third experiment allocates a fixed neuron budget in a network that must contain a narrow layer. We compare the profile that minimizes $\beta_{\text{eff}}$ against uniform-plus-bottleneck and DeepMAD-style profiles on CIFAR-10 and CIFAR-100, all with identical training budgets, and report test accuracy and seed variance. A win means higher median accuracy and lower seed-to-seed failure rate at equal neuron count.
The proposal fails cleanly if either of two things happens. The first is that $\kappa_\phi$ for orthogonal ReLU turns out to equal the Gaussian constant, meaning the nonlinearity alone restores Hanin's law. The second is that the $\beta$-matched profiles behave alike under both initializations. Either outcome would itself settle the open question of whether orthogonality changes the finite-width law for nonlinear networks.
References
- Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
- How to Start Training: The Effect of Initialization and Architecture
- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
- Initialization of ReLUs for Dynamical Isometry
- On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
- Vanishing Curvature in Randomly Initialized Deep ReLU Networks
- MetaInit: Initializing learning by learning to initialize
- DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network
- Dynamical Isometry for Residual Networks
- Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice