In the classical view of optimization, the learning rate is a "temperature" that should gradually cool down. As training progresses, we decrease the step size to allow the model to settle into a local minimum. However, a growing body of research suggests that this monotonic decay is often suboptimal. Instead, periodically increasing the learning rate—a technique known as cyclical learning rates or warm restarts—allows models to navigate complex loss landscapes more effectively, find flatter minima, and even produce ensembles for free.
The Energy to Escape Sharp Minima
The primary intuition behind nonmonotonic schedules is that the loss surfaces of deep neural networks are "bumpy," filled with numerous local optima of varying quality. A standard decaying learning rate might trap a model in the first local minimum it encounters. If this minimum is "sharp"—meaning the loss increases rapidly with small perturbations to the weights—the model is likely to generalize poorly to unseen data.
By periodically resetting the learning rate to a high value, as proposed in SGDR (Stochastic Gradient Descent with Warm Restarts) by Ilya Loshchilov, we provide the optimizer with the "kinetic energy" necessary to catapult out of these narrow, sharp wells. As the learning rate subsequently decays (often following a cosine curve), the model settles into a new region. This process repeats, effectively "shaking" the model until it finds a broader, flatter basin that is more robust to data shifts.
Navigating the Geometry of Loss Landscapes
Why do these flatter regions matter? Pavel Izmailov and colleagues demonstrated in Averaging Weights Leads to Wider Optima and Better Generalization that the optima found by standard SGD often lie on the periphery of wide, high-performing regions.
Nonmonotonic schedules facilitate a specific type of exploration:
- The Peak Phase: A high learning rate prevents the model from settling and forces it to traverse the "ridges" between valleys.
- The Cooling Phase: As the rate drops, the model slides down into a nearby valley.
- The Reset: Before it gets stuck forever, the rate jumps back up, and the search continues.
This cycle is particularly effective in constrained settings. For instance, Sihyeon Seong et al. in Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling showed that schedules peaking in the middle of training help models escape poorly generalizing regions that early, low-energy optimization might have favored.
Training One Model to Get an Ensemble for Free
One of the most powerful applications of this idea is Snapshot Ensembling (Snapshot Ensembles: Train 1, Get M for Free). Gao Huang and Geoff Pleiss realized that if a cyclical schedule forces a model to visit multiple distinct local minima, we can simply save a "snapshot" of the weights at the end of each cooling cycle.
Instead of training $M$ different models from scratch (which is computationally expensive), we train one model and collect $M$ sets of weights along its trajectory. Because the high-learning-rate spikes ensure these minima are diverse, the resulting ensemble significantly outperforms any single point on the path. This "trajectory ensembling" has been successfully adapted for tasks ranging from Salient Object Detection to Reinforcement Learning.
Beyond Simple Cycles: Adaptive and Structural Resets
The concept of "resetting" has expanded beyond just the learning rate. Researchers are now synchronizing these cycles with architectural changes:
- Weight Decay: Frank Hutter and Ilya Loshchilov showed in AdamW that decoupling weight decay from the gradient update is essential for cyclical schedules to work correctly with adaptive optimizers.
- Model Compression: When pruning or quantizing a network, the model suffers a "structural shock." Anish Acharya and Inderjit S Dhillon use cyclical schedules to help models recover from the loss of parameters by re-exploring the landscape for a new optimum (Online Embedding Compression).
- Layer Re-initialization: The RIFLE strategy (RIFLE: Backpropagation in Depth) by Haoyi Xiong periodically resets the final layer of a network while simultaneously spiking the learning rate, forcing the backpropagation of meaningful updates deep into the model's backbone.
Theoretical Grounding: Bandwidth and Spectral Bias
While these techniques often feel like "hacks," theoretical work is catching up. Ya-xiang Yuan and Mikael Johansson have proposed a "bandwidth-based" framework, proving that as long as the learning rate stays within a globally decaying "envelope," nonmonotonicity does not break convergence (Bandwidth-based Step-Sizes).
Furthermore, Gongyue Zhang suggests that cyclical schedules help manage "spectral bias"—the tendency of optimizers to learn different frequencies of data at different speeds. By oscillating the learning rate, we prevent the model from stagnating in a single "spectral regime," leading to faster and more balanced convergence.