REX: Revisiting Budgeted Training with an Improved Schedule

John Chen, Cameron R. Wolfe, Anastasios KyrillidisMLSys 2021 · OpenReview

TL;DR

This paper tackles the challenge of optimizing deep learning models under varying computational budgets by proposing a novel learning rate schedule called REX. REX consistently outperforms existing schedules across a wide range of budgets and tasks, offering a robust and easy-to-use solution for practitioners.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is computationally expensive, and practitioners often face a trade-off between model performance and available resources (time, money). This is particularly true when dealing with limited "budgets," meaning fewer training epochs are possible. Existing learning rate schedules, which are crucial for achieving good performance, often excel in either low-budget (few epochs) or high-budget (many epochs) scenarios, but rarely both. For instance, linear schedules are good for low budgets, while step schedules are better for high budgets. This forces practitioners to choose a schedule based on an often-unknown budget, leading to suboptimal results. The paper aims to bridge this gap by developing a learning rate schedule that performs well across the entire spectrum of training budgets.

Methods

The authors frame the learning rate schedule selection problem as a combination of two components:

  1. Profile: The underlying continuous function that defines the learning rate's decay curve (e.g., linear, exponential).
  2. Sampling Rate: How frequently the learning rate is updated based on the profile (e.g., every iteration, every 10% of epochs).

They analyze existing schedules through this lens and find that no single profile works optimally for all sampling rates. To address this, they propose the Reflected Exponential (REX) schedule. REX is designed to interpolate between a linear schedule and a "delayed" linear schedule (where the decay starts later), but crucially, it achieves this without introducing additional hyperparameters. The REX schedule's mathematical formulation is provided, showing how it adapts the learning rate based on the current training progress.

Datasets

The paper evaluates REX across a diverse set of standard datasets and tasks, including:

  • Image Classification: CIFAR10, CIFAR100, ImageNet, STL10
  • Object Detection: Pascal VOC
  • Natural Language Processing: GLUE benchmark (excluding WNLI)
  • Image Generation: MNIST

These datasets are used with various well-known model architectures like ResNet, VGG, Wide ResNet, and BERT.

Key Insights and Ideas

  • The Profile-Sampling Rate Dichotomy: Decomposing learning rate schedules into profiles and sampling rates provides a novel framework for understanding their behavior and limitations.
  • No Universal Optimal Profile: The paper demonstrates that different learning rate profiles are optimal for different sampling rates and budget regimes, highlighting the inadequacy of a one-size-fits-all approach.
  • The Power of Delayed Decay: While delaying the start of learning rate decay can improve performance, it introduces an extra hyperparameter. REX cleverly achieves similar benefits without this drawback.
  • REX's Budget-Agnostic Performance: The core insight is that REX consistently performs well across a wide range of training budgets, from very low (1% of epochs) to very high (100% of epochs), making it a highly practical choice.
  • Simplicity is Key: REX requires no additional computation, storage, or hyperparameters, making it easy to implement and adopt.

Key Results

The paper presents extensive empirical results across numerous experiments. Key findings include:

  • Consistent Superiority: REX consistently outperforms or matches state-of-the-art learning rate schedules (linear, step, cosine, OneCycle, etc.) across various datasets, models, and optimizers (SGD and Adam).
  • Low Budget Advantage: In low-budget regimes (e.g., 1-10% of total epochs), REX often shows significant improvements over other schedules, particularly the linear schedule.
  • High Budget Performance: Even in high-budget regimes, REX maintains competitive or superior performance, often surpassing schedules that are traditionally favored for longer training runs.
  • Broad Applicability: REX demonstrates strong performance across diverse tasks like image classification, object detection, image generation, and natural language processing. For example, in Table 4 (RN20-CIFAR10), REX achieves the best Top-1 performance at 1% epochs (27.94 ±.46) and also performs very well at 100% epochs (8.18 ±.15), outperforming the linear schedule (7.32 ±.14) and step schedule (7.52 ±.05) at the highest budget.

Limitations and Future Work

  • Hyperparameter Sensitivity: While REX itself has no extra hyperparameters, the paper acknowledges that learning rate schedules, in general, cannot compensate for a poorly chosen initial learning rate, as shown in Figure 4.
  • Computational Constraints: For some experiments, particularly with large models and datasets like RN50-ImageNet and BERT, the authors were limited to fewer runs or epochs due to computational constraints.
  • Future Work: The authors suggest that REX's success opens avenues for further research into profile-sampling rate combinations and their impact on other optimization aspects. They also highlight the potential for REX to be integrated into more complex training paradigms.

Summary updated

Abstract

Deep learning practitioners often operate on a computational and monetary budget. Thus, it is critical to design optimization algorithms that perform well under any budget. The linear learning rate schedule is considered the best budget-aware schedule, as it outperforms most other schedules in the low budget regime. On the other hand, learning rate schedules -- such as the \texttt{30-60-90} step schedule -- are known to achieve high performance when the model can be trained for many epochs. Yet, it is often not known a priori whether one's budget will be large or small; thus, the optimal choice of learning rate schedule is made on a case-by-case basis. In this paper, we frame the learning rate schedule selection problem as a combination of $i)$ selecting a profile (i.e., the continuous function that models the learning rate schedule), and $ii)$ choosing a sampling rate (i.e., how frequently the learning rate is updated/sampled from this profile). We propose a novel profile and sampling rate combination called the Reflected Exponential (REX) schedule, which we evaluate across seven different experimental settings with both SGD and Adam optimizers. REX outperforms the linear schedule in the low budget regime, while matching or exceeding the performance of several state-of-the-art learning rate schedules (linear, step, exponential, cosine, step decay on plateau, and OneCycle) in both high and low budget regimes. Furthermore, REX requires no added computation, storage, or hyperparameters.
BibTeX
@inproceedings{chen2021rex,
  author = {John Chen and Cameron R. Wolfe and Anastasios Kyrillidis},
  title = {{REX: Revisiting Budgeted Training with an Improved Schedule}},
  booktitle = {MLSys},
  year = {2021},
  url = {https://openreview.net/forum?id=CtbScb6-ey},
}
Figure