TL;DR
This paper introduces a simple, direct empirical method called "local branched training" to measure the Critical Batch Size (CBS) during language model pretraining. By tracking how CBS evolves, the authors demonstrate that a dynamic batch size warmup strategy can train an OLMo 1B model to a slightly better final loss while saving 43% of gradient steps.
What Problem Does This Paper Address?
When training large language models (LLMs), increasing the batch size allows for greater data parallelism, which speeds up wall-clock training time. However, if the batch size is too large, the model becomes token-inefficient, meaning it requires more total tokens to reach the same loss. The sweet spot is the Critical Batch Size (CBS): the maximum batch size up to which training efficiency is preserved.
Historically, researchers relied on a proxy called the gradient noise scale (popularized by OpenAI in 2018 and used in GPT-3) to estimate CBS. However, this proxy relies on fragile assumptions—namely, that the optimizer is standard SGD and the loss landscape is perfectly well-conditioned. In practice, modern LLMs are trained using the Adam optimizer on highly complex landscapes, making the gradient noise scale an unreliable metric. This paper addresses the lack of a reliable, assumption-free method to measure and leverage CBS in practical settings.
Methods
Instead of relying on theoretical proxies, the authors propose Local Branched Training to measure CBS directly:
- Branching: From a specific pretraining checkpoint, they spin off multiple short, parallel training runs ("branches").
- Scaling: Each branch uses a different batch size ($k \cdot B$) and scales the learning rate using the square-root scaling rule ($\sqrt{k} \cdot \eta$), which is theoretically principled for the Adam optimizer.
- Observation: They train these branches for a short window ($\Delta = 2\text{B}$ tokens) and observe the smoothed final loss.
- Thresholding: The CBS ($B^*$) is identified as the largest batch size that does not significantly degrade the loss compared to smaller batch sizes (within a small tolerance $\epsilon$).
Using the insights gained from how CBS changes over time, they implement a Batch Size Warmup strategy: starting with a small batch size at initialization and doubling it whenever the local CBS permits.
Datasets
The experiments were conducted using the open-source OLMo (1B and 7B) model frameworks.
- Pretraining Data: The models were trained on the Dolma corpus.
- Evaluation: The models were evaluated on held-out sets of C4 and The Pile, alongside a suite of downstream tasks (including ARC-Challenge, ARC-Easy, CommonsenseQA, HellaSwag, MMLU, and PIQA) evaluated via Bits-Per-Byte (BPB) metrics.
Key Insights and Ideas
- CBS is Dynamic, Not Fixed: Directly measuring CBS reveals that it starts near zero at initialization, increases rapidly during the first 50 billion tokens, and eventually plateaus.
- Model Size Independence: Crucially, the evolution of CBS looks remarkably similar for both the 1B and 7B model scales. This implies that researchers can run cheap, small-scale branched training to map out the CBS trajectory and use those insights to optimize much larger, expensive training runs.
- The Noise Scale Proxy Fails: Empirically comparing their direct method against the traditional gradient noise scale proxy showed that the old method underestimates the true CBS by several orders of magnitude and fails to capture the correct qualitative trend for larger models.
Key Results
The authors validated their batch size warmup strategy by training an OLMo 1B model from scratch up to 608 billion tokens, comparing it against a small-batch control and a fixed large-batch control.
- 43% Fewer Gradient Steps: The batch size warmup schedule achieved a 43% reduction in sequential gradient steps compared to the small-batch control.
- Superior Loss: Not only did the warmup strategy save massive compute time, but it also achieved a slightly better final pretraining loss (2.5891) than both the small-batch control (2.6057) and the fixed large-batch control (2.5962).
- Downstream Performance: Downstream task evaluations confirmed that the compute savings did not come at the cost of model capability; performance remained broadly competitive or superior across the board.
Limitations and Future Work
While highly effective, the local branched training method introduces two new hyperparameters: the token window size ($\Delta$) and the loss tolerance ($\epsilon$). The authors noted that their batch size doubling schedule was chosen via manual inspection of offline plots.
Future work includes automating this process into an online adaptive batch size scheduler that can dynamically measure CBS and adjust the batch size on the fly during a single pretraining run, as well as relaxing the constraint of only changing batch sizes by powers of two.
Summary updated