TL;DR
This paper tackles the challenge of optimizing deep learning models under varying computational budgets by proposing a novel learning rate schedule called REX. REX consistently outperforms existing schedules across a wide range of budgets and tasks, offering a robust and easy-to-use solution for practitioners.
What Problem Does This Paper Address?
Training deep neural networks (DNNs) is computationally expensive, and practitioners often face a trade-off between model performance and available resources (time, money). This is particularly true when dealing with limited "budgets," meaning fewer training epochs are possible. Existing learning rate schedules, which are crucial for achieving good performance, often excel in either low-budget (few epochs) or high-budget (many epochs) scenarios, but rarely both. For instance, linear schedules are good for low budgets, while step schedules are better for high budgets. This forces practitioners to choose a schedule based on an often-unknown budget, leading to suboptimal results. The paper aims to bridge this gap by developing a learning rate schedule that performs well across the entire spectrum of training budgets.
Methods
The authors frame the learning rate schedule selection problem as a combination of two components:
- Profile: The underlying continuous function that defines the learning rate's decay curve (e.g., linear, exponential).
- Sampling Rate: How frequently the learning rate is updated based on the profile (e.g., every iteration, every 10% of epochs).
They analyze existing schedules through this lens and find that no single profile works optimally for all sampling rates. To address this, they propose the Reflected Exponential (REX) schedule. REX is designed to interpolate between a linear schedule and a "delayed" linear schedule (where the decay starts later), but crucially, it achieves this without introducing additional hyperparameters. The REX schedule's mathematical formulation is provided, showing how it adapts the learning rate based on the current training progress.
Datasets
The paper evaluates REX across a diverse set of standard datasets and tasks, including:
- Image Classification: CIFAR10, CIFAR100, ImageNet, STL10
- Object Detection: Pascal VOC
- Natural Language Processing: GLUE benchmark (excluding WNLI)
- Image Generation: MNIST
These datasets are used with various well-known model architectures like ResNet, VGG, Wide ResNet, and BERT.
Key Insights and Ideas
- The Profile-Sampling Rate Dichotomy: Decomposing learning rate schedules into profiles and sampling rates provides a novel framework for understanding their behavior and limitations.
- No Universal Optimal Profile: The paper demonstrates that different learning rate profiles are optimal for different sampling rates and budget regimes, highlighting the inadequacy of a one-size-fits-all approach.
- The Power of Delayed Decay: While delaying the start of learning rate decay can improve performance, it introduces an extra hyperparameter. REX cleverly achieves similar benefits without this drawback.
- REX's Budget-Agnostic Performance: The core insight is that REX consistently performs well across a wide range of training budgets, from very low (1% of epochs) to very high (100% of epochs), making it a highly practical choice.
- Simplicity is Key: REX requires no additional computation, storage, or hyperparameters, making it easy to implement and adopt.
Key Results
The paper presents extensive empirical results across numerous experiments. Key findings include:
- Consistent Superiority: REX consistently outperforms or matches state-of-the-art learning rate schedules (linear, step, cosine, OneCycle, etc.) across various datasets, models, and optimizers (SGD and Adam).
- Low Budget Advantage: In low-budget regimes (e.g., 1-10% of total epochs), REX often shows significant improvements over other schedules, particularly the linear schedule.
- High Budget Performance: Even in high-budget regimes, REX maintains competitive or superior performance, often surpassing schedules that are traditionally favored for longer training runs.
- Broad Applicability: REX demonstrates strong performance across diverse tasks like image classification, object detection, image generation, and natural language processing. For example, in Table 4 (RN20-CIFAR10), REX achieves the best Top-1 performance at 1% epochs (27.94 ±.46) and also performs very well at 100% epochs (8.18 ±.15), outperforming the linear schedule (7.32 ±.14) and step schedule (7.52 ±.05) at the highest budget.
Limitations and Future Work
- Hyperparameter Sensitivity: While REX itself has no extra hyperparameters, the paper acknowledges that learning rate schedules, in general, cannot compensate for a poorly chosen initial learning rate, as shown in Figure 4.
- Computational Constraints: For some experiments, particularly with large models and datasets like RN50-ImageNet and BERT, the authors were limited to fewer runs or epochs due to computational constraints.
- Future Work: The authors suggest that REX's success opens avenues for further research into profile-sampling rate combinations and their impact on other optimization aspects. They also highlight the potential for REX to be integrated into more complex training paradigms.
Summary updated