Stacking as Accelerated Gradient Descent

Naman Agarwal, Pranjal Awasthi, Satyen Kale, Eric ZhaoarXiv 2024 · DOI · arXiv

Introduction

Training state-of-the-art deep learning models, particularly large Transformers, is staggeringly expensive, often costing millions of dollars in compute. To mitigate this, researchers use "greedy layer-wise pretraining" or "stacking." In this approach, training begins with a shallow network which is progressively deepened by adding layers.

A common "stacking" trick is to initialize these new layers by copying the weights of the layers immediately below them. While this works remarkably well in practice, the underlying mechanism has remained poorly understood. Previous explanations relied on heuristic assumptions about layers being "few-shot learners." This paper provides a rigorous, optimization-based explanation: stacking is mathematically equivalent to Nesterov’s Accelerated Gradient Descent (AGD).

The Stacking Mechanism

The core idea of stacking involves a multi-stage training process. As shown in Figure 1, a model starts with a single block. After a period of training, a new block is added, initialized by copying the parameters of the existing block. This process repeats, effectively growing the model's depth while leveraging previously learned features.

Figure 1: The stage-wise stacking process where Transformer blocks are copied to initialize new layers.

Theoretical Framework: Stacking as Momentum

The authors analyze stacking through the lens of functional gradient descent—viewing optimization as moving through a space of functions rather than just a list of parameters. They compare three initialization strategies for adding a new component to an ensemble:

  1. Zero Initialization: Recovers standard Gradient Descent.
  2. Random Initialization: Recovers Stochastic Gradient Descent on a "smoothed" version of the loss function.
  3. Stacking Initialization: By copying the previous layer's weights, the update rule for the next stage naturally incorporates a term representing the difference between the current and previous states $(F_t - F_{t-1})$. This is the hallmark of momentum, specifically Nesterov’s AGD.

The "aha moment" is realizing that copying weights effectively "bakes in" the velocity of the optimization. In an additive model (like boosting), stacking exactly recovers Nesterov's AGD. For deep residual networks, the authors provide a formal proof for Deep Linear Residual Networks, showing that Nesterov's acceleration is robust enough to persist even when the architecture doesn't perfectly match the textbook AGD formula.

Empirical Validation

The paper validates its theory using both synthetic deep linear networks and real-world models like BERT Base. In the BERT experiments, the authors compared stacking against random initialization over four stages of training (675,000 total steps).

Figure 2: Log perplexity comparison between stacking and random initialization during BERT training.

As seen in Figure 2, stacking initialization led to a significantly lower log perplexity compared to random initialization at every stage boundary. This empirical success aligns with the theoretical prediction of accelerated convergence. For smooth losses, stacking improves the convergence rate from $O(T^{-1})$ to $O(T^{-2})$. For strongly convex losses, it improves from $\exp(-\Omega(T/\kappa))$ to $\exp(-\Omega(T/\sqrt{\kappa}))$.

Key Insights and Limitations

  • Unifying Boosting and Deep Learning: The framework treats classical Gradient Boosting and modern ResNets/Transformers as two sides of the same coin, showing that the same acceleration principles apply to both.
  • The β Parameter: Introducing a trainable "momentum" parameter (β) to scale the stacked output further improved performance, reaching a log perplexity of 1.75 compared to the standard 1.77.
  • Linearity Constraint: While the functional analysis suggests the principle holds for non-linear networks, the most rigorous proofs are currently limited to linear residual networks. Formalizing this for the non-linear case remains a significant open challenge.
  • Warm-up Requirements: The theory assumes the model starts in a "good" region. In practice, stacking usually requires a short "warm-up" phase of standard training before the acceleration benefits become apparent.

By providing a formal link between a common engineering heuristic and a fundamental optimization algorithm, this work offers a principled path toward more efficient training of massive neural architectures.

Summary updated

Abstract

Stacking, a heuristic technique for training deep residual networks by progressively increasing the number of layers and initializing new layers by copying parameters from older layers, has proven quite successful in improving the efficiency of training deep neural networks. In this paper, we propose a theoretical explanation for the efficacy of stacking: viz., stacking implements a form of Nesterov's accelerated gradient descent. The theory also covers simpler models such as the additive ensembles constructed in boosting methods, and provides an explanation for a similar widely-used practical heuristic for initializing the new classifier in each round of boosting. We also prove that for certain deep linear residual networks, stacking does provide accelerated training, via a new potential function analysis of the Nesterov's accelerated gradient method which allows errors in updates. We conduct proof-of-concept experiments to validate our theory as well.
BibTeX
@misc{agarwal2024stacking,
  author = {Naman Agarwal and Pranjal Awasthi and Satyen Kale and Eric Zhao},
  title = {{Stacking as Accelerated Gradient Descent}},
  year = {2024},
  eprint = {2403.04978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.04978},
}
Figure