TL;DR
This paper introduces a novel two-phase learning rate adjustment strategy to bridge the gap between large-batch training's scalability and small-batch training's superior generalization. By extending the initial "exploration" phase even after the training loss plateaus, and then adaptively decaying the learning rate based on the loss landscape, they achieve near small-batch accuracy with significantly larger batch sizes and minimal computational overhead.
What Problem Does This Paper Address?
Training large neural networks efficiently often relies on data parallelism, where increasing the batch size is a natural way to scale up computation. However, a well-known issue is that large batch sizes tend to degrade the model's generalization performance. This is attributed to the reduced noise in the gradients, which can cause the model to converge to sharp local minima that don't perform as well on unseen data. While various methods exist to mitigate this, the impact of learning rate decay schedules on large-batch generalization has been largely overlooked. This paper specifically investigates how to optimally decay the learning rate to achieve both scalability and accuracy.
Methods
The core contribution is a two-phase adaptive learning rate adjustment framework:
- Phase 1: Exploration Phase: This phase begins with a large learning rate. Crucially, the paper argues that decaying the learning rate solely based on a flattened training loss is detrimental. Instead, they propose extending this "exploration" phase even after the loss appears to have plateaued. This allows the model to explore the parameter space more broadly and improve the underlying margin distribution (a proxy for generalization performance). The duration of this phase is guided by a heuristic: if a loss plateau is detected, the model trains for a fraction of the total epochs (e.g., 1/3) before moving to the next phase.
- Phase 2: Convergence Phase: Once sufficient exploration has occurred, the framework switches to a convergence phase. Here, the learning rate is adaptively decayed. This decay is Hessian-aware, meaning it considers the curvature of the loss landscape. Specifically, it uses the maximum Hessian eigenvalue ($h_{max}$) and the training loss ($f'(w)$) to decide when and how much to decay the learning rate. If the loss increases and the Hessian eigenvalue also increases (indicating a move towards a sharper minimum), the learning rate is decayed. The Hessian eigenvalue is approximated efficiently using power iteration on the largest output-side layer.
Datasets
The proposed framework was evaluated on the following standard benchmark datasets:
- CIFAR-10
- CIFAR-100
- SVHN
- ImageNet
Key Insights and Ideas
- Loss Plateau is Misleading for Generalization: The paper's central insight is that a flattened training loss in large-batch settings doesn't necessarily mean training progress has stopped. The model can still improve its generalization (margin distribution) even when the loss isn't decreasing.
- Extended Exploration is Key: Allowing the model to train for more epochs with a high learning rate, even after the loss plateaus, significantly benefits generalization. This "exploration" phase is critical for finding flatter minima.
- Hessian-Aware Decay for Convergence: Adapting the learning rate decay based on the loss landscape's curvature (using Hessian information) helps guide the model towards flatter, more generalizable minima during the convergence phase.
- Scalability Without Sacrificing Accuracy: The proposed method demonstrates that it's possible to achieve accuracies very close to small-batch training while using batch sizes orders of magnitude larger, offering substantial computational benefits.
Key Results
- On CIFAR-10 with ResNet-20, the proposed "two-phase" method achieved 92.66% accuracy with a batch size of 8,192, which is very close to the small-batch (batch size 128) accuracy of 92.83%.
- The framework significantly improved upon the standard "large-batch" (linear scaling rule) performance across all tested datasets.
- Combining the "two-phase" framework with Layer-wise Adaptive Rate Scaling (LARS) further boosted performance, showing the framework's compatibility with other large-batch techniques.
- The Hessian approximation and adaptive decay added only a 3-5% computational overhead per epoch, making it practically efficient.
Limitations and Future Work
- Hyperparameter Sensitivity: While the paper proposes heuristics for the exploration phase length and suggests a decay factor ($\beta \approx 0.5$), the optimal settings might still require some tuning. Table 6 shows that deviating too much from the suggested exploration phase length can slightly decrease accuracy.
- Epoch Budget: The effectiveness of the exploration phase is tied to the total epoch budget. If the budget is too small, an extended exploration phase might leave insufficient time for convergence. The paper suggests that finding a "proper epoch budget" is important.
- Future Work: The authors suggest harmonizing their framework with other generalization-focused optimization techniques as a promising direction. They also highlight the potential for this approach to enable even larger-scale deep learning solutions.
Summary updated