Introduction
In production machine learning, data often arrives continuously. Intuitively, when updating a neural network with today's new data, you should initialize it with the weights from yesterday's model—a process known as "warm-starting"—rather than training from scratch.
While warm-starting works perfectly for convex problems like logistic regression and saves massive amounts of compute, it surprisingly harms the generalization of deep neural networks. Even though a warm-started neural network will reach the same training accuracy as a randomly initialized one, it performs noticeably worse on unseen test data. Because of this, practitioners are often forced to retrain models entirely from scratch to maintain high accuracy, incurring enormous financial and environmental costs. This paper investigates why warm-starting hurts deep learning generalization and proposes a simple, effective fix.
The Warm-Starting Pathology
The authors systematically demonstrate the warm-start generalization gap across various architectures and optimizers using standard image classification benchmarks (CIFAR-10, CIFAR-100, and SVHN).
They find that warm-starting damages generalization almost immediately: even just a few epochs of training on a subset of data is enough to permanently degrade the final model's test accuracy if those weights are used as an initialization for the full dataset.
In an online learning setting where new samples are added sequentially, standard warm-starting trains much faster than random initialization, but the generalization gap widens significantly as more data is added. If you tune hyperparameters (like the learning rate) aggressively enough to close this generalization gap, the model essentially "forgets" its initialization. At that point, it takes just as long to train as a randomly initialized model, defeating the purpose of the warm start.
The authors identify gradient imbalance as the culprit. In warm-started models, gradients from the newly added data are much larger in magnitude than gradients from the old data. This imbalance hinders optimization and prevents the model from learning a generalizable hypothesis.
The "Shrink and Perturb" Solution
To fix this, the authors introduce the Shrink, Perturb, Repeat trick. When new data arrives, instead of directly using the previous weights $\theta$, they initialize the new training round by shrinking the weights toward zero and adding a small amount of Gaussian noise:
$$ \theta_{new} = \lambda \theta_{old} + p $$
where $\lambda$ is a shrinkage factor (e.g., $0 < \lambda < 1$) and $p \sim \mathcal{N}(0, \sigma^2)$.
Shrinking the weights increases the model's loss and entropy without changing its fundamental decision boundaries (especially in ReLU networks). This artificially boosts the gradients of the old data, balancing them with the new data and allowing the model to learn effectively from the combined dataset.
Key Results
- The Generalization Gap: On CIFAR-10 using a ResNet-18 and Adam optimizer, a randomly initialized model trained on 100% of the data achieves 78.0% accuracy. A model trained on 50% of the data and warm-started on 100% drops to 74.4% accuracy.
- Shrink-Perturb Success: Using the Shrink and Perturb trick ($\lambda = 0.6$, noise scale = $0.01$) completely closes this gap. The model achieves the same accuracy as a randomly initialized model but trains in a fraction of the wall-clock time.
- Bonus Regularization: In some pre-training scenarios, the shrink-perturb trick even slightly outperforms random initialization due to an implicit regularization effect caused by the noise injection.
Limitations and Future Work
The authors note that the exact point at which warm-starting becomes detrimental compared to random initialization depends heavily on the statistical properties of the specific dataset, which cannot be easily predicted. While the shrink-perturb trick avoids the need to predict this crossover, the paper primarily evaluates the method on standard computer vision benchmarks. Future work could explore how this trick scales to massive language models, reinforcement learning, or other domains where continuous online learning and pre-training are heavily relied upon.
Summary updated