Deep fully connected networks have a known size rule. To train reliably, their hidden layers must get wider as the network gets deeper. The precise statement is that random fluctuations in signal size grow with $\beta = \sum_l 1/n_l$, the sum over layers of one over each layer's width $n_l$. If $\beta$ is large, the gradient for any single random draw of the weights is either tiny or enormous. Hanin and Rolnick argue that this is an architectural problem that no choice of weight distribution can fix. Deep linear networks contradict the spirit of that claim: with orthogonal weights (random rotations rather than independent Gaussian entries), the width needed to train no longer depends on depth at all. This proposal asks whether that escape carries over to real nonlinear ReLU networks, and if so, for how long.
The core idea is simple. A "looks-linear" initialization pairs each ReLU unit with a mirror unit and ties their weights, so the whole ReLU network computes an exactly orthogonal linear map at step zero. At that moment the width-depth law is void: signal norms are preserved exactly, with zero fluctuation, whatever the widths. Training then breaks the tie, and the network gradually starts to behave like a randomly gated ReLU network again. We conjecture that the width-depth law does not disappear but is deferred. It returns in proportion to how far each layer has drifted from linearity. That drift can be measured, so the conjecture makes quantitative predictions about which width schedules, depths and learning rates succeed.
Core Claim
Define $\varepsilon_l(t) \in [0,1]$ as the fraction of layer $l$'s output variance at training step $t$ carried by the non-linear component. For a mirrored unit pair, this is the part of the output proportional to $|z|$ rather than $z$, where $z$ is the pre-activation. We propose an effective reciprocal width
$$\beta_{\mathrm{eff}}(t) = \sum_l \frac{\varepsilon_l(t)}{n_l}.$$
We conjecture that the across-seed variance of the log squared gradient norm at step $t$ is approximately $\beta_{\mathrm{eff}}(t)$. The form comes from Hanin's log-normal law, under which the log-norm variance is $\beta$ when every unit is an independent coin-flip gate. Units still in their linear state should contribute no such variance. Two limiting cases check the formula. Gaussian He initialization has $\varepsilon_l = 1$ and recovers ordinary $\beta$. Looks-linear initialization starts at $\beta_{\mathrm{eff}} = 0$.
The claim has three consequences. First, the trainability boundary in the depth-versus-width plane should be vertical early in training and should bend toward the diagonal $n \propto L$ only as $\varepsilon$ grows. Second, bottleneck layers, which inflate $\beta$, should be harmless at first and costly later. Third, slowing the growth of $\varepsilon$ in the narrowest layers should buy back most of the width that the law demands.
Relation to Prior Work
Three papers motivate the gap.
- Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients? proves that gradient variance is exponential in $\beta$ for ReLU networks with independent weights. It explicitly excludes dependent weight structures, so it does not contradict our claim. It also leaves that case open.
- How to Start Training separates two failure modes. The first is a drifting mean activation size, which initialization can fix. The second is exponential variance, which it calls architectural. Its proof also assumes independent Gaussian or uniform weights.
- Products of Many Large Random Matrices and Gradients in Deep Neural Networks supplies the log-normal law we reuse. It holds only at initialization and only under independent half-probability gating.
Our $\varepsilon$-weighting is a hypothesis about what replaces that gating once units are tied and then partly untied.
Three papers supply the method, and each stops where we start.
- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks shows that a vertical boundary holds throughout training, but only for linear networks of uniform width.
- Initialization of ReLUs for Dynamical Isometry introduces the mirrored block $W = [W_0, -W_0; -W_0, W_0]$ with orthogonal $W_0$. It tests depths only up to 10 and gives no width-depth bound.
- Dynamical Isometry for Residual Networks builds residual blocks with the same looks-linear structure (called RISOTTO) and trains them without normalization.
None of these three tracks how quickly the linear state erodes, or how that erosion interacts with non-uniform widths.
Three papers constrain the design.
- On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization proves that orthogonal and Gaussian initializations are equivalent in the infinite-width "lazy" limit, where weights barely move. Any advantage must therefore be a finite-width, finite-$L/n$ effect, which is exactly the regime we study.
- The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry shows that even perfect isometry leaves the largest curvature eigenvalue growing linearly with depth. The stable learning rate must then shrink roughly as $1/L$. We therefore tune learning rates per depth, so that a curvature effect is not mistaken for a fluctuation effect.
- The Future is Log-Gaussian shows that vanilla ReLU ResNets have exploding correlation-driven variance at depth-to-width ratio 1. A random-sign "balanced" variant reduces that factor from about 250 to about 10. This is an independent, cheaper symmetry-based fix that we must compare against.
Research Program
The theoretical part has two pieces.
The first is a perturbative analysis of a mirrored-pair ReLU layer whose two outgoing weight blocks differ by a small matrix $\Delta_l$. To leading order this layer is an orthogonal map plus a rectified term of relative size $\|\Delta_l\|$. Composing such layers should give log-norm variance $\sum_l \varepsilon_l/n_l$ to first order. The calculation would be the inverse-width bookkeeping of the products-of-random-matrices paper, restricted to the untied subspace.
The second piece describes how $\varepsilon_l(t)$ grows under gradient descent. We suspect, without proof, that early growth is roughly linear in the learning rate times the step count. If so, the effective "safe training window" before the width law re-engages scales as $n_{\min}/(\eta L)$, where $n_{\min}$ is the narrowest width and $\eta$ the learning rate.
Empirically, we build an instrumented training harness that logs $\varepsilon_l$ per layer, gradient norms, and loss across many seeds. We map trainability phase diagrams over depth, width and width schedule for each initialization arm.
Baselines
The naive reference is a plain ReLU MLP or CNN with Gaussian He initialization. Following the warning in How to Start Training, it uses a correctly scaled untruncated normal rather than the library truncated normal. This arm defines the diagonal boundary that $\beta$ predicts.
The second arm is Haar-orthogonal initialization with ordinary ReLU. The Xiao et al. ReLU-isometry paper argues it fails because ReLU zeroes half the signal. Including it separates "orthogonal" from "tied and linear."
The third arm is the strongest explicit-structure competitor. For plain networks this is MetaInit from MetaInit: Initializing learning by learning to initialize. MetaInit reports 3.7% test error on a 28-layer plain CNN on CIFAR-10, against 6.0% for the same network with batch normalization. It is the strongest result in the file for training deep unnormalized plain networks. For residual networks the competitor is RISOTTO, which reports 60.63±0.28% CIFAR-100 accuracy without batch normalization, against 58.25±1.64% for Fixup. We rerun both under our protocol rather than relying only on the quoted numbers.
The fourth arm is the balanced random-sign trick from The Future is Log-Gaussian.
The fifth arm is a matched-width tanh network with orthogonal weights, the smooth-activation isometry route.
All arms share the same data, epochs and per-depth learning-rate sweep, with an equal number of sweep points per arm. Each arm uses both SGD and Adam.
Beating the strongest competitor means one of two things:
- Reaching MetaInit's or RISOTTO's accuracy at measurably smaller width for the same depth.
- Matching it at the same width while showing a boundary shift that $\beta_{\mathrm{eff}}$ predicts and MetaInit's curvature criterion does not.
The key ablation is a "delayed untying" variant. It holds mirrored pairs exactly tied for the first $K$ steps by projecting the update back onto the tied subspace. It then releases them, either uniformly or only in the widest layers. If $\beta_{\mathrm{eff}}$ is the operative quantity, keeping only the narrow layers tied should recover most of the benefit at a fraction of the expressivity cost.
Why not just…?
Why not just add residual connections and batch normalization? They do control variance. The batch-normalization Gram-matrix paper (On Bridging the Gap between Mean Field and Finite Width) shows the error stops growing with depth, and How to Start Training shows scaled residual branches avoid the variance failure. But this changes the question rather than answering it. The question is whether initialization alone can lift an architectural law. Batch normalization also has known costs: MetaInit beats it on the 28-layer plain CNN, and RISOTTO's authors note it interferes with adversarial training. Vanilla residual networks still fail at depth-to-width ratio 1, according to the log-Gaussian paper. We report batch-normalized networks as an upper reference, not as a competitor to our mechanism.
Why not just use Adam? This is a real threat. Vanishing Curvature in Randomly Initialized Deep ReLU Networks trains 128-layer ReLU MLPs with Adam where SGD stalls. Adam rescales vanishing gradients, but rescaling cannot undo across-seed log-normal spread once activations overflow, which How to Start Training documents. Our phase diagrams are run with Adam as well. If Adam with Gaussian initialization already has a vertical boundary, the practical case for our proposal shrinks to SGD and to the bottleneck experiments, and we will say so.
Why not just make every layer wider? Uniform width minimizes $\beta$ for a fixed neuron budget, by Hanin's power-mean argument. Width costs compute quadratically in a dense layer, however, and “No Free Lunch” in Neural Architectures? shows deep narrow networks are more expressive per parameter. The measurable payoff of our proposal is the width saved at fixed depth and accuracy. If that saving is negligible, the alternative wins.
Why not just use the random-sign balanced trick? It is cheaper than tying weights and removes inter-layer correlation. However, it still uses independent gates, so by Hanin's theorem it should leave the $\beta$ dependence intact. If the balanced trick matches looks-linear initialization on the bottleneck test, our account is wrong.
Proposed Experiments
Phase diagrams. We train MLPs on vectorized MNIST and Fashion-MNIST with depths from 10 to 500 and widths from 32 to 1024. The trainability metric is epochs to 20% accuracy, the metric used in Hanin's and Hanin–Rolnick's papers, which report no tabular values in the file. We record the fraction of seeds that diverge or stall and fit the slope of the boundary for each arm. The prediction is that the looks-linear boundary is vertical within the safe window $n_{\min}/(\eta L)$ and bends afterward. Gaussian He should be diagonal throughout.
Bottleneck test. Pairs of networks share depth and total neuron count but differ in $\beta$ by up to a factor of four, through one narrow layer or a tapering schedule. Gaussian He should separate them from step zero. Looks-linear initialization should show no difference until $\varepsilon$ in the narrow layer grows. The gap should then track $\beta_{\mathrm{eff}}$.
Quantitative fit. Across about 50 seeds we regress the variance of the log squared gradient norm at each logged step on $\beta_{\mathrm{eff}}(t)$. A slope near one supports the formula. A poor fit, or a fit no better than raw $\beta$, refutes it.
Transfer to convolutional and residual networks. We use a 28-layer plain CNN on CIFAR-10 against MetaInit's reported 3.7% error, and BN-free residual networks on CIFAR-100 against RISOTTO's reported 60.63%. Each is rerun at the original width and at half and quarter width. The proposal succeeds if looks-linear plain networks with delayed untying keep accuracy at reduced width where the other arms degrade.
Falsification. Any of the following would falsify the central claim:
- The looks-linear boundary is already diagonal early in training.
- Delayed untying in only the narrow layers does not move the boundary.
- The balanced random-sign variant performs equally well on the bottleneck test.
References
- Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
- How to Start Training: The Effect of Initialization and Architecture
- Products of Many Large Random Matrices and Gradients in Deep Neural Networks
- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
- Initialization of ReLUs for Dynamical Isometry
- Dynamical Isometry for Residual Networks
- On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
- The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
- The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization
- MetaInit: Initializing learning by learning to initialize
- On Bridging the Gap between Mean Field and Finite Width Deep Random Multilayer Perceptron with Batch Normalization
- Vanishing Curvature in Randomly Initialized Deep ReLU Networks
- “No Free Lunch” in Neural Architectures? A Joint Analysis of Expressivity, Convergence, and Generalization