As neural networks grow deeper, optimizing them becomes notoriously difficult. When information travels through dozens of parameterized layers, gradients can vanish, explode, or become completely uncorrelated—a phenomenon known as "shattered gradients." As David Balduzzi et al. demonstrate in The Shattered Gradients Problem: If resnets are the answer, then what is the question?, standard random initializations cause gradients to resemble white noise as depth increases, making effective optimization nearly impossible.
The core intuition behind overcoming this bottleneck is remarkably simple: make the network behave like an identity mapping at the start of training. If a network initially passes its input directly to its output without distortion, the gradient signal can propagate perfectly across extreme depths. This provides a smooth optimization landscape for stochastic gradient descent (SGD) to begin learning before the network gradually introduces non-linear transformations.
An early realization of this principle can be found in Highway Networks. Rupesh Kumar Srivastava et al. showed in Training Very Deep Networks that initializing gating units with a negative bias forces the network to initially function as a sequence of identity mappings, allowing researchers to train networks with hundreds of layers directly via gradient descent.
Simplifying the Landscape in Residual Networks
Residual Networks (ResNets) naturally incorporate identity paths through physical skip connections ($y = x + F(x)$). However, if the parameterized branch $F(x)$ outputs large, random values at initialization, it can still disrupt gradient flow and complicate the initial optimization landscape.
To maximize the benefit of the shortcut, the parameterized blocks should be forced to "get out of the way" on day one. As proven by Moritz Hardt et al. in Identity Matters in Deep Learning, initializing weights near zero in a residual architecture ensures the network begins training as a pure identity transformation. This positioning avoids the problematic critical points associated with traditional deep architectures.
Mathematically, this strategy creates a strong prior that favors the shortcut path during early training. Tianyi Liu et al. show in Towards Understanding the Importance of Shortcut Connections in Residual Networks that this prevents the model from starting in regions of the loss landscape prone to spurious local optima. Furthermore, Difan Zou et al. demonstrate in On the Global Convergence of Training Deep Linear ResNets that initializing residual weights to zero allows gradients to flow perfectly, enabling global convergence with a network width that is completely independent of depth.
Standard Block (Random Init): x ---> [ Layer ] ---> Noise + x (Shattered Gradients)
Identity-Biased Block: x ---> [ Layer=0 ] -> 0 + x (Perfect Gradient Flow)
This concept translates into highly practical training adjustments. For instance, Tong He et al. in Bag of Tricks for Image Classification with Convolutional Neural Networks introduce a widely adopted tweak: initializing the scaling parameter ($\gamma$) of the final batch normalization layer in a residual block to zero. This simple change forces the entire residual block to act as an identity mapping at the start of training, reducing initial network complexity and facilitating cleaner gradient flow.
Even in dynamic architectures where layers can choose to halt computation early to save energy, this principle holds. In Spatially Adaptive Computation Time for Residual Networks, Michael Figurnov initializes halting biases to negative values. This forces the network to utilize more layers initially, ensuring all units receive sufficient gradient signal before the model begins optimizing for computational efficiency.
Mimicking Shortcuts in Plain Networks
While physical skip connections solve the optimization problem, they introduce memory bandwidth overhead because the hardware must keep the residual tensor in memory while computing the block's transformations. Can we achieve the training stability of a ResNet in a "plain" sequential network without physical shortcuts?
By baking the identity mapping directly into the initial weight matrices, we can simulate the behavior of residual shortcuts. Guillaume Berger et al. leverage this in QuickSRNet: Plain Single-Image Super-Resolution Architecture for Faster Inference on Mobile Platforms. By initializing convolutional weights to approximate an identity mapping, their plain network stabilizes gradient flow and matches the accuracy of residual architectures while running significantly faster on mobile hardware.
Similarly, Guilin Li et al. introduce an identity-mimicking initialization in Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts that uses a Dirac delta distribution to set the starting weights of a convolutional network, allowing plain networks to be trained effectively from scratch.
This strategy applies across various constrained parameter spaces and alternative architectures:
- Structured Linear Layers: When replacing standard fully-connected layers with highly constrained, memory-efficient alternatives, Marcin Moczulski et al. show in ACDC: A Structured Efficient Linear Layer that diagonal components must be initialized near the identity matrix with minimal noise to break symmetry and ensure stable convergence.
- Orthogonal States: George Philipp et al. show in Gradients explode - Deep Networks are shallow - ResNet explained that setting a network into an approximately orthogonal state mimics linear behavior, allowing standard architectures to achieve training performance comparable to ResNets.
- Logic Gate Networks: Even in non-traditional, logic-based networks, Lukas Rüttgers et al. demonstrate in Light Differentiable Logic Gate Networks that initializing logic gates to function as pass-through identities prevents vanishing gradients, delaying the learning of complex boolean transformations until a stable signal is established.
Theoretical Foundations and the Linear Bias
Why does starting near the identity matrix work so well from an optimization standpoint?
Theoretical analysis reveals that identity initialization acts as an implicit accelerator. Sanjeev Arora et al. demonstrate in On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization that using identity initialization in deep linear chains prevents the vanishing gradients typically associated with near-zero initializations, acting as a built-in preconditioner that speeds up gradient descent.
However, this strategy introduces a specific geometric bias. Peter Bartlett et al. prove in Gradient descent with identity initialization efficiently learns positive definite linear transformations that initializing deep linear networks at the identity matrix biases the optimization process toward learning positive definite transformations. This ensures efficient convergence to a global optimum, provided the target mapping does not require flipping the sign of any eigenvalues. This provides a strong inductive bias: it forces the network to focus training efforts on learning the residual error signal relative to a base linear transformation, a property also exploited in engineering applications like digital predistortion by Yibo Wu et al. in Residual Neural Networks for Digital Predistortion.
This aligns with a broader discovery by Lechao Xiao et al. in The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks: during the early stages of training, high-capacity neural networks naturally behave almost exactly like simple linear models.
To perfectly align a network with this neutral starting state, researchers have developed "zero-output" initialization variants. For instance, Amit Daniely in Neural Networks Learning and Memorization with (almost) no Over-Parameterization pairs neurons with identical weights but opposite output signs. This design choice ensures the network initially outputs zero, simplifying the optimization landscape for SGD.
Taking this further, Simone Bombari et al. show in Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization that such symmetric weight initializations allow gradient descent to proceed from a mathematically neutral state while maintaining a well-conditioned Jacobian. This guarantees a linear rate of convergence while avoiding the massive over-parameterization traditionally required by deep learning theory.
Key Foundational Work
Training Very Deep Networks (2015) — contributed 2 concept elements to this direction.
ACDC: A Structured Efficient Linear Layer Marcin Moczulski (2016) — contributed 1 concept elements to this direction.