In the early days of deep learning, training networks with many hidden layers was notoriously difficult. Researchers found that as they added depth, standard gradient-based optimization often stalled, leaving the network trapped in poor local minima or struggling with "vanishing gradients" where the signal for learning failed to reach the earliest layers.
The breakthrough that revitalized the field was the realization that a deep network does not need to be trained all at once. Instead, it can be built "greedily"—one layer at a time—using unsupervised learning to create a robust foundation before any labels are even introduced.
The Optimization Bottleneck
Before 2006, the prevailing wisdom was that deep architectures were fundamentally more expressive than shallow ones but practically untrainable. As Yoshua Bengio and Olivier Delalleau argued in On the Expressive Power of Deep Architectures, deep circuits can represent complex functions much more efficiently than shallow ones, yet the non-convex objective functions of deep networks made random initialization a recipe for failure.
The core problem is that a randomly initialized deep network starts in a "chaotic" state. When we try to teach it a specific task (like identifying a digit) using backpropagation, the gradients must travel through many layers of random weights. By the time the gradient reaches the first layer, it is often too diluted to provide meaningful updates.
Building Hierarchies Greedily
The solution, pioneered by Geoffrey E. Hinton in What kind of graphical model is the brain?, was to treat each layer as an independent unsupervised module. Instead of training the whole stack at once, we train the first layer to reconstruct its input. Once that layer is "stable," we fix its weights and use its outputs as the training data for the next layer.
This "greedy" approach typically uses one of two building blocks:
- Restricted Boltzmann Machines (RBMs): Generative models that learn to capture the statistical distribution of the input data.
- Stacked Autoencoders (SAEs): Networks trained to compress and then reconstruct the input, forcing the hidden layers to extract the most salient features.
By stacking these modules, the network learns a hierarchy of features. The first layer might learn simple edges, the second learns shapes, and the third learns complex parts. Crucially, this happens without any labels. As Christian Schmid and Santiago Jaramillo note in Passive exposure to task-relevant stimuli enhances categorization learning, this unsupervised phase "primes" the system, building latent representations that make the final task much easier to learn.
Pre-training as a Better Starting Point
Why does this help? The intuition is that unsupervised pre-training places the network's weights in a region of the parameter space that is already "aware" of the data's structure.
In Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure, Ruslan Salakhutdinov and Geoffrey E. Hinton demonstrated that this initialization allows the network to model the "data manifold"—the underlying shape of the data—before labels are introduced. This acts as a powerful regularizer. Instead of the network trying to learn everything from scratch using a few labeled examples, it only needs to learn how to map its existing, high-quality features to the specific labels.
This is particularly vital in fields where labeled data is scarce but raw data is plentiful. For example, in medical imaging, Dinggang Shen and colleagues found that unsupervised structural learning prevents models from stalling when trained on small clinical datasets Deep Learning in Medical Image Analysis.
From Generative to Discriminative
Once the stack of layers is pre-trained, the final step is fine-tuning. We add a final output layer (like a softmax layer for classification) and use standard backpropagation to adjust all the weights in the network.
Because the network starts from a stable, representative state rather than a random one, the fine-tuning process converges much faster and reaches better solutions. This transition was a watershed moment for practical applications. Dong Yu and Frank Seide used this exact two-stage procedure—unsupervised pre-training followed by supervised fine-tuning—to achieve a massive leap in accuracy for speech recognition at Microsoft Conversational Speech Transcription Using Context-Dependent Deep Neural Networks.
Generalization and Robustness
Beyond just making deep networks "trainable," pre-training makes them more robust. In Deep Learners Benefit More from Out-of-Distribution Examples, Yoshua Bengio showed that layer-wise pre-training helps networks generalize better to "messy" or out-of-distribution data. By learning to reconstruct corrupted inputs (as in Denoising Autoencoders), the network develops a hierarchy of robust features that are less sensitive to noise.
While modern techniques like Batch Normalization and Residual Connections have made it possible to train very deep networks without explicit layer-wise pre-training, the core intuition remains central to the field: the most efficient way to learn a complex task is to first understand the world in which that task exists.