To train a modern neural network in a reasonable timeframe, researchers rely on massive parallelism. By spreading the workload across thousands of GPUs, we can process more data at once—increasing the "batch size." In an ideal world, doubling the batch size would halve the number of training steps required to reach a target accuracy, leading to a linear speedup.
However, deep learning scaling is not a free lunch. As we increase the batch size, we eventually hit a point of diminishing returns known as the Critical Batch Size (CBS). Beyond this threshold, adding more data to a single update doesn't significantly improve the gradient's quality; it just wastes electricity.
The U-Shaped Curve of Computational Cost
The fundamental challenge of large-batch training is balancing the number of iterations with the work done per iteration. Hideaki Iiduka describes this as a minimization of "Stochastic First-order Oracle (SFO) complexity"—the total number of individual gradient evaluations required to converge.
If you visualize the relationship between batch size and total computational cost, it forms a U-shaped curve:
- Small Batch Regime: The batch is too small and noisy. You spend too many steps wandering around the loss landscape, making the total cost high.
- The Sweet Spot (CBS): This is the bottom of the U. Here, the batch is large enough to provide a stable gradient signal but small enough that every sample still contributes meaningful new information.
- Large Batch Regime: You have exceeded the CBS. While the number of steps might still decrease slightly, the cost per step (in terms of compute and memory) grows faster than the benefits. You are effectively calculating the same gradient over and over with redundant data.
Naoki Sato and Hideaki Iiduka have demonstrated that this critical threshold exists even for complex architectures like Generative Adversarial Networks (GANs) Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule.
Why the Wall Exists: Gradient Noise and Correlation
To understand why scaling fails, we have to look at what a batch actually does. A batch provides an estimate of the "true" gradient of the entire dataset. Gradient correlation is a key ingredient to accelerate SGD with momentum suggests that the efficiency of increasing batch size is constrained by how much new information each sample provides. If the gradients from different data points are highly correlated, a larger batch quickly becomes redundant.
Furthermore, Andrew Cheng and Kiwon Lee argue that there is a mathematical transition point, which they link to an "Implicit Conditioning Ratio." Below this point, momentum doesn't actually speed up training; above it, you enter an "accelerated" regime where the batch is large enough for momentum to effectively smooth out the trajectory Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions.
Scaling Laws for the Critical Batch Size
A common misconception is that larger models automatically require larger batches. Recent work by Hanlin Zhang and Depen Morwani has shown that the CBS scales primarily with the amount of training data, not the number of model parameters How Does Critical Batch Size Scale in Pre-training?.
As we move toward "long-horizon" training on trillions of tokens, the CBS actually increases. This is good news for hardware scaling: it means that as our datasets grow, we can use even larger GPU clusters efficiently. However, the CBS is not a static number. William Merrill and Shane Arora found that the CBS grows rapidly during the early stages of training before plateauing Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training. This justifies the common practice of "batch size warmup," where the batch size is gradually increased as training progresses.
Overcoming Instability at Scale
Even if we stay below the CBS, large batches introduce a new problem: optimization divergence. When the batch size is huge, the learning rate must also be large to maintain progress. But a high learning rate can cause the model to "explode" or settle into poor local minima.
To combat this, researchers have developed specialized optimizers:
- LARS (Layer-wise Adaptive Rate Scaling): Developed by Igor Gitman and Boris Ginsburg, LARS adjusts the learning rate for each layer independently based on the ratio of weights to gradients. This allowed scaling ImageNet training to a batch size of 32,000 Large Batch Training of Convolutional Networks.
- LAMB: An evolution of LARS that enabled Tong Wang to train object detectors on the COCO dataset in just 12 minutes Large Batch Optimization for Object Detection: Training COCO in 12 minutes.
- Muon: A newer optimizer that treats parameters as matrices rather than vectors. Naoki Sato has shown that Muon has its own unique CBS, potentially allowing for even more efficient large-scale training than standard first-order methods Analysis of Muon's Convergence and Critical Batch Size.
Summary
The Critical Batch Size represents the fundamental limit of data parallelism. It is the boundary where the physics of the optimization landscape—the noise in the data and the curvature of the loss—prevents us from simply throwing more hardware at the problem. By understanding how CBS scales with data and training time, and by using adaptive optimizers to maintain stability, researchers can push the boundaries of exascale deep learning without hitting the efficiency wall.