The Optimization Barrier in Deep Networks
In theory, adding more layers to a neural network should always increase or at least maintain its expressive power. If the extra layers in a deeper network simply perform an identity mapping, the deeper model should produce the exact same training error as its shallower counterpart. However, experiments historically revealed a frustrating paradox: as networks grew deeper, training accuracy would saturate and then sharply degrade.
This degradation is not caused by overfitting, but by a fundamental optimization barrier. As networks grow deeper, propagating signals across dozens of parameterized layers becomes incredibly difficult. Aoran Wang notes that standard deep architectures suffer severely from the vanishing gradient problem, where error signals shrink exponentially as they backpropagate through repeated layer multiplications. As emphasized by Xin Zhang et al. and Vikash Singh et al., this optimization failure prevents deep networks from learning complex, high-dimensional representations effectively, causing performance to collapse purely due to the inability to train them.
The Residual Solution: Learning Corrections
The core insight of residual learning is to fundamentally change what a layer is tasked with learning. In a standard feedforward network, a stack of layers is forced to directly approximate an underlying target mapping, $H(x)$. Residual learning reconfigures this setup: we let the layers approximate a residual mapping, $F(x) = H(x) - x$. The original input $x$ is then added back to the output of the layers via an identity shortcut connection (or skip connection), yielding the final transformation:
$$H(x) = F(x) + x$$
+---------+
| x |--------+
+---------+ |
| |
[Weight Layer] | Identity
| | Shortcut
(ReLU) |
| |
[Weight Layer] |
| |
V V
F(x) + x <----+
|
(ReLU)
|
V
Why does this mathematical rearrangement make optimization so much easier?
As the authors of Interpretable 3D Human Action Analysis with Temporal Convolutional Networks explain, this structure creates an additive identity mapping. Instead of forcing each layer to generate a completely new feature map from scratch, each successive layer only needs to learn a refinement or incremental correction to the input. Carl R. Andersson et al. describe this as modeling incremental corrections to an identity mapping. If the optimal transformation is closer to an identity mapping than to a brand-new feature space, it is significantly easier for the optimization algorithm to drive the weights of $F(x)$ toward zero than to learn an identity mapping from scratch using standard non-linear layers.
The Gradient Highway
The most powerful property of the identity shortcut becomes apparent when we look at how gradients flow backward during training. In a standard feedforward network, the gradient must pass sequentially through every single parameterized layer, getting multiplied by weight matrices at each step. If those weights are poorly scaled, the gradient vanishes before it can reach the earliest layers of the network.
With an identity shortcut, the gradient has a direct, unattenuated pathway to flow backward. When computing the derivative of the output $H(x) = F(x) + x$ with respect to its input $x$, the chain rule yields:
$$\frac{\partial H}{\partial x} = \frac{\partial F}{\partial x} + 1$$
The crucial term here is the $+ 1$. Even if the gradient through the weight layers ($\frac{\partial F}{\partial x}$) approaches zero due to vanishing weights or activations, the clean $+1$ term ensures that the error signal propagates all the way back to the earliest layers without attenuation. This "gradient highway" stabilizes backpropagation, allowing networks to scale to hundreds or even thousands of layers while maintaining stable convergence.
Preserving Signal and Feature Integrity
Beyond stabilizing gradients, skip connections fundamentally alter how information is preserved across a network's depth. In deep architectures without shortcuts, low-level features and raw signal components tend to wash out or disappear as they are repeatedly transformed by non-linear operations.
Samuel Horváth et al. identify this as a critical issue in deep network operations, demonstrating that residual connections act as a stabilizing mechanism that prevents the total disappearance of features by providing paths for low-order signal components to bypass deep layers entirely. Similarly, Rongqiang Guan et al. observe that residual connections allow the integration of raw signal data directly with high-level abstract features. By allowing low-level details to skip intermediate processing, the network can preserve fine-grained spatial or temporal details across deep network hierarchies, combining raw inputs with highly abstract concepts at the final layers.
Theoretical Efficiency and Approximation Power
This architectural choice is not just an empirical trick; it possesses rigorous theoretical advantages over standard feedforward networks. Chenghao Liu et al. demonstrate that identity shortcuts allow the persistence of input signals across layers without consuming any tunable parameters. This enables the network's weights to focus exclusively on learning the non-linear residual transformation rather than wasting capacity on preserving the identity mapping.
From an approximation standpoint, Dmitry Yarotsky proved that these deep, information-passing pathways provide the exact structural requirements needed to achieve optimal function approximation rates. By decoupling the depth of the network from the degradation of the signal, residual learning allows neural networks to fully exploit their depth, making them fundamentally more efficient function approximators than traditional architectures.
Key Foundational Work
Improved Music Genre Classification with Convolutional Neural Networks (2016) — contributed 2 concept elements to this direction.