Abstract
Finding the optimal learning rate (LR) is a critical yet resource-intensive challenge in pretraining foundation models. Traditional methods, such as scaling laws and proxy model tuning, often fail to generalize or require significant computational overhead. This paper introduces AdaLRS (Adaptive Learning Rate Search), a "plug-and-play" algorithm designed to identify the optimal LR within a single training run. By leveraging the convexity of loss descent velocity, AdaLRS dynamically adjusts the LR to maximize progress, significantly reducing training costs while maintaining or improving model performance.
Introduction
In the development of Large Language Models (LLMs) and Vision Language Models (VLMs), the learning rate is arguably the most influential hyperparameter. An improperly tuned LR—either too high or too low—can lead to training instability, catastrophic collapse, or inefficient convergence. Currently, researchers rely on two primary strategies: Scaling Laws, which predict LR based on model size but often falter with architectural changes, and Proxy Tuning, which involves training numerous small models to estimate the behavior of a larger one. Both approaches are costly and indirect.
AdaLRS addresses this by asking a fundamental question: Can we find the optimal learning rate dynamically during the training of the target model itself?
Methodology
The Shared Optimum Hypothesis
The core of AdaLRS is the theoretical discovery that training loss and the velocity of loss descent are both convex functions of the learning rate. Crucially, these functions share the same optimum. This implies that the learning rate which produces the fastest instantaneous drop in loss is also the learning rate that will lead to the best final model performance.
The AdaLRS Algorithm
AdaLRS operates as a wrapper around existing schedulers (e.g., Cosine or WSD) and follows a four-stage process:
- Monitoring: The algorithm tracks the slope of the loss curve over a window of $k$ steps using a least-squares fit to determine the current "velocity."
- Upscaling: If the velocity begins to decay, AdaLRS attempts to "kickstart" progress by increasing the LR.
- Validation & Backtracking: To prevent instability, AdaLRS employs a safety net. After an LR adjustment, it evaluates the new velocity. If the loss spikes or progress stalls, the algorithm restores the model and optimizer states to their previous values and attempts a downscaled LR instead.
- Decay: The intensity of the search (scaling factors) is gradually reduced over time to ensure stability as the model approaches convergence.
Experimental Setup
The researchers validated AdaLRS across diverse pretraining environments:
- LLM Pretraining: Utilizing Qwen2.5 (1.5B and 7B) architectures trained on the SlimPajama dataset (up to 160B tokens).
- VLM Pretraining: Utilizing the SAIL-VL (2B) architecture, trained on detailed captions and OCR-heavy datasets.
- Evaluation: Performance was measured via downstream benchmarks including Alpaca-Gen, KNIGHT-Gen, DocVQA, and TextVQA.
Key Insights and Results
Efficiency and Cost Reduction
AdaLRS demonstrated the ability to move suboptimal initial LRs into the optimal neighborhood rapidly. For models starting with a conservative (too small) LR, AdaLRS reached the baseline loss in approximately 50% of the training steps, effectively reducing total training costs by 30% to 50%.
Performance Gains
In experiments with the 7B LLM, starting with a suboptimal LR and applying AdaLRS resulted in a final training loss of 2.38, compared to 2.74 for the baseline. This translated to superior perplexity and benchmark performance, proving that the algorithm does not just speed up training but can actually find better local minima.
Robustness
The algorithm proved effective across different model scales (1.5B to 7B) and training paradigms, including both from-scratch pretraining and continual pretraining. Because it performs state restoration (backtracking), it is uniquely capable of "undoing" bad steps that would otherwise corrupt model weights in standard adaptive optimizers.
Limitations and Future Work
While AdaLRS is highly effective, it is sensitive to initialization. If the initial LR is so high that it causes catastrophic forgetting in the very first steps, the backtracking mechanism may not be able to recover the model. Therefore, the authors recommend starting with a conservative LR and allowing AdaLRS to scale up.
Future research will focus on refining the search granularity to pinpoint the exact optimal LR and exploring the sensitivity of the algorithm's internal hyperparameters, such as window size and scaling factors, although current results suggest the system is relatively robust to these settings.
Conclusion
AdaLRS offers a robust, efficient, and "plug-and-play" solution to the learning rate search problem. By focusing on loss descent velocity and incorporating a state-restoration safety net, it allows researchers to bypass expensive proxy tuning and achieve optimal training efficiency in a single run.
Summary updated