Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training

William Merrill, Shane Arora, Dirk Groeneveld, Hannaneh Hajishirzi

TL;DR

This paper introduces a simple, direct empirical method called "local branched training" to measure the Critical Batch Size (CBS) during language model pretraining. By tracking how CBS evolves, the authors demonstrate that a dynamic batch size warmup strategy can train an OLMo 1B model to a slightly better final loss while saving 43% of gradient steps.

What Problem Does This Paper Address?

When training large language models (LLMs), increasing the batch size allows for greater data parallelism, which speeds up wall-clock training time. However, if the batch size is too large, the model becomes token-inefficient, meaning it requires more total tokens to reach the same loss. The sweet spot is the Critical Batch Size (CBS): the maximum batch size up to which training efficiency is preserved.

Historically, researchers relied on a proxy called the gradient noise scale (popularized by OpenAI in 2018 and used in GPT-3) to estimate CBS. However, this proxy relies on fragile assumptions—namely, that the optimizer is standard SGD and the loss landscape is perfectly well-conditioned. In practice, modern LLMs are trained using the Adam optimizer on highly complex landscapes, making the gradient noise scale an unreliable metric. This paper addresses the lack of a reliable, assumption-free method to measure and leverage CBS in practical settings.

Methods

Instead of relying on theoretical proxies, the authors propose Local Branched Training to measure CBS directly:

  1. Branching: From a specific pretraining checkpoint, they spin off multiple short, parallel training runs ("branches").
  2. Scaling: Each branch uses a different batch size ($k \cdot B$) and scales the learning rate using the square-root scaling rule ($\sqrt{k} \cdot \eta$), which is theoretically principled for the Adam optimizer.
  3. Observation: They train these branches for a short window ($\Delta = 2\text{B}$ tokens) and observe the smoothed final loss.
  4. Thresholding: The CBS ($B^*$) is identified as the largest batch size that does not significantly degrade the loss compared to smaller batch sizes (within a small tolerance $\epsilon$).

Using the insights gained from how CBS changes over time, they implement a Batch Size Warmup strategy: starting with a small batch size at initialization and doubling it whenever the local CBS permits.

Datasets

The experiments were conducted using the open-source OLMo (1B and 7B) model frameworks.

  • Pretraining Data: The models were trained on the Dolma corpus.
  • Evaluation: The models were evaluated on held-out sets of C4 and The Pile, alongside a suite of downstream tasks (including ARC-Challenge, ARC-Easy, CommonsenseQA, HellaSwag, MMLU, and PIQA) evaluated via Bits-Per-Byte (BPB) metrics.

Key Insights and Ideas

  • CBS is Dynamic, Not Fixed: Directly measuring CBS reveals that it starts near zero at initialization, increases rapidly during the first 50 billion tokens, and eventually plateaus.
  • Model Size Independence: Crucially, the evolution of CBS looks remarkably similar for both the 1B and 7B model scales. This implies that researchers can run cheap, small-scale branched training to map out the CBS trajectory and use those insights to optimize much larger, expensive training runs.
CBS over training for OLMo 1B and 7B. The trend is highly consistent across scales.
  • The Noise Scale Proxy Fails: Empirically comparing their direct method against the traditional gradient noise scale proxy showed that the old method underestimates the true CBS by several orders of magnitude and fails to capture the correct qualitative trend for larger models.

Key Results

The authors validated their batch size warmup strategy by training an OLMo 1B model from scratch up to 608 billion tokens, comparing it against a small-batch control and a fixed large-batch control.

  • 43% Fewer Gradient Steps: The batch size warmup schedule achieved a 43% reduction in sequential gradient steps compared to the small-batch control.
  • Superior Loss: Not only did the warmup strategy save massive compute time, but it also achieved a slightly better final pretraining loss (2.5891) than both the small-batch control (2.6057) and the fixed large-batch control (2.5962).
  • Downstream Performance: Downstream task evaluations confirmed that the compute savings did not come at the cost of model capability; performance remained broadly competitive or superior across the board.
Batch size schedule, learning rate schedule, and training loss comparison.

Limitations and Future Work

While highly effective, the local branched training method introduces two new hyperparameters: the token window size ($\Delta$) and the loss tolerance ($\epsilon$). The authors noted that their batch size doubling schedule was chosen via manual inspection of offline plots.

Future work includes automating this process into an online adaptive batch size scheduler that can dynamically measure CBS and adjust the batch size on the fly during a single pretraining run, as well as relaxing the constraint of only changing batch sizes by powers of two.

Summary updated

Abstract

The right batch size is important when training language models at scale: a large batch size is necessary for fast training, but a batch size that is too large will harm token efficiency. To navigate this tradeoff, McCandlish et al. (2018) suggest that a critical batch size (CBS), below which training will not substantially degrade loss, can be estimated based on the gradient noise scale during training. While their method has been adopted in practice, e.g., when training GPT-3, strong assumptions are required to justify gradient noise as a proxy for the CBS, which makes it unclear whether their approach should be trusted in practice, limiting its applicability. In this paper, we introduce a simple, empirical approach to directly measure the CBS and show how the CBS evolves over training. Applying our approach to the OLMo models, we find that CBS is near 0 at initialization, increases rapidly at first, and then plateaus as training progresses. Furthermore, we find that this trend holds across different model sizes (1B and 7B), suggesting CBS from small training runs can inform larger-scale training runs. Our findings about how the CBS changes over training motivate batch size warmup as a natural way to reliably train language models at large batch size: start the batch size small and increase it as the CBS grows. To validate this claim, we use batch size warmup to train OLMo 1B to slightly better loss than the original training run with 43% fewer gradient steps. This shows how our framework can be applied to reliably train language models at larger batch sizes, increasing data parallelism without compromising performance.
BibTeX
@inproceedings{merrill2025critical,
  author = {William Merrill and Shane Arora and Dirk Groeneveld and Hannaneh Hajishirzi},
  title = {{Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=XUKUx7Xu89},
}
Figure