When training deep neural networks, we rarely rely on raw, instantaneous gradients. Instead, we smooth them out using running averages to stabilize trajectories and accelerate convergence. However, a subtle mathematical artifact lurks at the very beginning of this process: because these running averages must start from somewhere, they are typically initialized at zero. This "zero-start" creates a severe initialization bias, artificially dragging early estimates toward the origin.
Over the past decade, understanding, correcting, and occasionally questioning the mechanisms that counteract this initialization bias has proven essential for stabilizing everything from early convolutional networks to modern large language models.
The Mechanics of the Zero-Start Bias
To understand why initialization bias occurs, consider the standard Exponential Moving Average (EMA) used to track a sequence of gradients $g_t$ at timestep $t$:
$$v_t = \beta v_{t-1} + (1 - \beta)g_t$$
If we initialize the accumulator at the origin ($v_0 = 0$) and unroll this recursion for the first few steps, we see that:
$$v_t = (1 - \beta) \sum_{i=1}^t \beta^{t-i} g_i$$
If we were to take a true weighted average of the gradients observed so far, the coefficients should sum to $1$. However, if we sum the coefficients in the equation above, we get:
$$(1 - \beta) \sum_{i=1}^t \beta^{t-i} = 1 - \beta^t$$
Because $\beta$ is typically close to $1$ (for instance, $\beta = 0.999$ for second moments in Adam), the term $1 - \beta^t$ is incredibly small during the first few iterations. As a result, the magnitude of $v_t$ is severely suppressed.
To counteract this, Kingma et al. in their foundational paper, Adam: A Method for Stochastic Optimization, introduced the now-ubiquitous bias correction factor. By dividing the raw estimate $v_t$ by $1 - \beta^t$, the optimizer scales the moving average back up to its true expected value, ensuring that early updates are not artificially muted.
The Dual Nature of Bias Correction: Warm-up vs. Explosion
What happens if we omit this correction? The consequences depend heavily on how the first and second moments interact, leading to two distinct failure modes or behaviors in deep learning pipelines.
1. Oversized and Destructive Updates
In adaptive optimizers like Adam, both the first moment (momentum) and the second moment (uncentered variance) are tracked via EMAs. If an implementation mistakenly omits the bias correction terms, the step size can become wildly unstable.
As Zhang et al. demonstrate in Revisiting Few-sample BERT Fine-tuning, omitting bias correction leads to oversized, erratic weight updates during the initial steps of training. In low-data regimes or few-sample fine-tuning, these early erratic steps can permanently derail the model's convergence, causing catastrophic degeneracy before the moving averages have time to collect enough data to self-correct.
2. Implicit Learning Rate Warm-up
Conversely, when properly implemented, the interaction of the first- and second-moment corrections provides a highly beneficial stabilization effect. Andriushchenko et al. show in On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines that re-enabling Adam's bias correction acts as an implicit learning rate warmup.
Because the second-moment correction scales differently than the first-moment correction early on, it naturally dampens the effective step size during the first few iterations. This implicit warmup stabilizes the initial phase of fine-tuning, preventing the model from falling into poor local minima and significantly reducing sensitivity to random seeds. This empirical benefit aligns with the theoretical analysis by Barakat et al. in Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization, which proves that the bias correction mechanism is mathematically critical for improving initial parameter estimates when the algorithm has no prior gradient history.
The Cold-Start Problem in Dynamic Networks
The necessity of bias correction becomes even more pronounced when neural network architectures change dynamically during training. Consider dynamic sparse training, where zeroed-out weights are suddenly re-activated mid-training to shift the model's capacity.
Global Timestep: t = 1000
[Existing Parameter] ---> Uses 1 - β^1000 ≈ 1 (Stable)
[Newly Woken Parameter] -> Accumulator = 0, but uses 1 - β^1000 (Explosion!)
As Xiao et al. discover in Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling, standard adaptive optimizers create a severe "cold-start" problem for these newly activated parameters. If the optimizer uses the global timestep $t$ to calculate the bias correction factor $1 - \beta^t$, it assumes the parameter has been accumulating gradients since step one. For a parameter woken up at step 1000, $1 - \beta^{1000}$ is nearly $1$. However, its actual internal accumulator is zero.
This mismatch causes the optimizer to skip scaling up the fresh accumulator, resulting in an update that is disproportionately large relative to its actual history, triggering catastrophic loss spikes. The authors resolve this by resetting the local timestep of newly activated parameters, ensuring the bias correction factor is calibrated to their actual lifespan rather than the global clock.
Generalizing Beyond Standard Gradients
The core intuition of correcting zero-start initialization bias extends far beyond standard gradient descent. It manifests whenever recursive filtering or averaging is applied to non-stationary signals in deep learning.
- Trend-Corrected Optimization: When optimizers incorporate higher-order momentum—such as tracking the acceleration or "trend" of gradients using Holt’s linear trend method—the initialization bias compounds. Zhou et al. demonstrate in On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods that specific, multi-layered bias correction terms are required to keep level and trend estimates accurate before the system reaches a steady state.
- Differentially Private Filtering: In privacy-preserving optimization, gradients are often heavily corrupted by injected noise. To clean this signal, Zhang et al. introduce a low-pass filter in DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction. Because recursive filters suffer from temporal lag and initialization bias, they apply a dedicated bias correction factor to ensure that the filtered gradient estimates remain accurate and responsive during the critical early phases of private training.
- Iterate and Weight Averaging: Bias correction is equally vital when averaging model weights rather than gradients. Standard Exponential Moving Averages of model weights often suffer from a performance lag because the early terms in the average include poorly optimized weights from the start of training. To eliminate this delay, Block et al. design EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes. Their algorithm, BEMA, applies a bias-correction term based on the model's displacement from its initial weights, capturing the stability of weight averaging without the historical lag.
Is Bias Correction Always Necessary?
Given its widespread adoption, it is easy to view bias correction as an absolute requirement. However, recent work has begun to map out the exact boundaries where this mathematical correction ceases to provide empirical value.
In Simplifying Adam: Bias Correction Debunked, Laing et al. challenge the conventional wisdom. They show that when practitioners employ optimal hyperparameters and explicit, well-tuned learning rate schedules (such as linear warmup followed by cosine decay), the implicit warmup provided by bias correction becomes redundant. Under these conditions, removing bias correction entirely yields no discernible loss in performance, suggesting that the complexity it introduces can be safely bypassed if the external training schedule is engineered correctly.
Furthermore, removing parts of the bias correction can drastically simplify the mathematical analysis of these complex algorithms. For instance, Défossez et al. show in A Simple Convergence Proof of Adam and Adagrad that removing the first-moment bias correction ensures that the optimizer's effective step size remains strictly monotonic throughout training. This structural simplification allows for a much cleaner, more elegant convergence proof in non-convex settings while still preserving the vital stability provided by the second-moment normalization.
Ultimately, whether achieved through explicit parameter centering at initialization—as explored by Melchior et al. in How to Center Deep Boltzmann Machines—or through dynamic rescaling factors, managing the transition from an uninitialized zero-state to a steady-state data stream remains a foundational pillar of successful deep learning optimization.
Key Foundational Work
Adam: A Method for Stochastic Optimization (2015) — contributed 3 concept elements to this direction.
How to Center Deep Boltzmann Machines (2016) — contributed 1 concept elements to this direction.