Predicting the Critical Batch Size with a Refined Mean Field Correction

Anyone training a large model must choose how many examples go into each gradient step. Larger batches cut the number of steps, but past a point, called the critical batch size, they waste data. The best current evidence is that this point is not fixed: it starts near zero and grows during training. Today it can only be measured by forking the run at checkpoints and training short side branches at several batch sizes, which is what Critical Batch Size Revisited did to justify a batch-size warmup. We propose to predict what those branches would show from statistics the main run already produces, and to decide online when to grow the batch.

The prediction comes from queueing theory. A mean field approximation replaces a noisy system of many interacting parts with its deterministic average. A refined mean field approximation adds the leading correction for finite size, computed by solving a small matrix equation (a Lyapunov equation, which gives the stationary covariance of fluctuations around a fixed point). Stochastic gradient descent (SGD) fits this picture. Gradient flow is the deterministic limit, and the finite batch size $B$, together with the step size $\eta$, sets how far the system sits from that limit. Our claim is specific. For Adam-style optimizers at the large learning rates used in practice, the "noise floor" is the extra loss a given batch size forces on the optimizer. It is predicted much better by the discrete-time, preconditioned refined correction than by the gradient noise scale that the continuous-time picture yields. That better prediction is enough to reproduce the measured growth of the critical batch size from a single run.

Core Claim

A Refined Mean Field Approximation shows that expected steady-state quantities differ from the deterministic limit by a structured $O(1/N)$ term, where $N$ is the system size. It computes that term from a Lyapunov equation $AW + WA^T + Q = 0$, with $A$ the Jacobian of the drift and $Q$ the noise covariance. This drops relative error at $N=10$ from over 20% to under 1%. On the Approximation Error of Mean-Field Models supplies the underlying fact that steady-state errors scale as $O(1/N)$ rather than $O(1/\sqrt{N})$.

We transfer this machinery to SGD, where the effective system size is $B/\eta$. Near a quasi-stable point with Hessian $H$, the dynamics involve a diagonal preconditioner $P$ (Adam's inverse square-root second moments), a per-example gradient covariance $\Sigma$, and step size $\eta$. The update is discrete and synchronous: every coordinate moves at once. That matches the discrete-time setting of A refined mean field approximation of synchronous discrete-time population models, not the continuous one. The fluctuation covariance $W$ therefore solves the discrete Lyapunov equation

$$W = (I - \eta P H)\, W\, (I - \eta P H)^T + \frac{\eta^2}{B} P \Sigma P ,$$

and the expected excess loss from finite batch size is $F(B) = \tfrac12 \mathrm{tr}(H W)$.

In a coordinate where the preconditioner is $p_i$, the curvature is $h_i$, and the gradient variance is $\sigma_i^2$, this has a closed form:

$$F(B) \approx \frac{\eta}{2B} \sum_i \frac{p_i \sigma_i^2}{2 - \eta p_i h_i} .$$

The continuous-time limit drops the $\eta p_i h_i$ term. It then gives $\eta \sum_i p_i \sigma_i^2 / 4B$, which is independent of curvature. That is essentially the gradient-noise-scale heuristic. The refined discrete formula differs in exactly the regime modern training occupies: directions near the edge of stability, where $\eta p_i h_i$ approaches 2 and the floor grows sharply. This explains why Critical Batch Size Revisited found static noise-scale proxies unreliable for modern optimizers. It also yields a testable prediction: the critical batch size should rise as the sharpest preconditioned directions approach stability limits.

Research Program

The first part is derivation. We will state the $1/B$ expansion for preconditioned discrete-time SGD, including the mean-shift term that the refined approximation also produces. Third derivatives of the loss displace the average iterate away from the minimum. This is the second half of the Gast–Van Houdt correction and is usually ignored in SGD analyses.

Training is not at equilibrium, so we will not assume a global fixed point. The refined theory assumes a unique attractive fixed point, and the file flags this as a known limitation. Instead we use the two-timescale view of Bias and Refinement of Multiscale Mean Field Models. High-curvature directions equilibrate within a few hundred steps and carry the noise floor. Low-curvature directions drift slowly and carry the remaining progress. The correction is applied only to the fast directions.

The second part is estimation at scale. The full Lyapunov solve costs $O(d^3)$, which is impossible for billions of parameters. We use a diagonal solve built from Adam's existing moment buffers and Hutchinson-style diagonal curvature estimates. We add an exact low-rank solve in the top-$k$ preconditioned Hessian directions, found by a few Lanczos iterations. Those top directions are exactly where the discrete factor $1/(2-\eta p h)$ departs from 1. This localized, low-rank treatment borrows the spirit of the localized covariance preconditioning in Fokker–Planck Particle Systems for Bayesian Inference, applied here to data noise rather than ensemble size.

The third part is a decision rule. At each check, we compare two quantities over the next window $\Delta$ tokens:

  • the predicted loss reduction from doubling $B$, which halves the fast-direction floor $F(B)$;
  • the predicted progress lost in the slow directions from taking half as many steps.

We double the batch when the first exceeds the second. The rule uses Merrill et al.'s square-root learning-rate scaling so that it is directly comparable to their warmup. Batch sizes stay at powers of two, which addresses the discrete-hardware constraint the file raises for control-based schedules.

Baselines

The naive reference is a fixed batch size with the same learning-rate schedule, run at both the small and large batch sizes. Critical Batch Size Revisited reports final pretraining losses for OLMo-1B of 2.6057 (fixed small batch) and 2.5962 (fixed large batch).

The strongest empirical competitor is that paper's batch-size warmup, calibrated by branched training. It reports a final loss of 2.5891 with 43% fewer gradient steps than the fixed-batch baseline. It is strongest because its schedule is tuned against direct measurements of the quantity we are trying to predict. We will re-run the branched protocol ourselves on FineWeb-Edu (FineWeb) at 150M and 1B parameters. The warmup gets the same token budget, optimizer, peak learning rate, and hardware as our rule. The compute spent on branches is reported separately, not hidden.

Beating it means matching its final loss within seed noise without any branch compute. An alternative win is lower loss at equal total compute once branch compute is counted.

The strongest analytic competitor is the closed-form schedule of Towards joint scaling laws with optimal batch size schedules. It derives a batch schedule from convex theory given the learning-rate schedule shape and data budget. It reports a 6–15% compute-efficiency gain and 62.5% fewer iterations on Llama3-7B; we cannot reproduce 7B and will compare at 1B. We run it with its prescribed schedule for our token budget and learning-rate shape.

The strongest online adaptive competitor is a gradient-noise-scale / signal-to-noise trigger in the style of Statistical Tests for Optimization Efficiency. It gets the same checking frequency and a threshold tuned with the same number of trials as our rule's single tolerance.

The key ablations isolate the proposed ingredients:

  • continuous-time Lyapunov correction: removes the $\eta p h$ term;
  • diagonal-only solve: no top-$k$ directions;
  • no multiscale split: applies the floor to all coordinates.

The continuous-time ablation is the decisive one, since it collapses our method to a curvature-free noise scale.

On the synthetic benchmarks we compare against Batch size selection by stochastic optimal control. That paper derives batch schedules from a continuous-SDE control problem assuming known Hessian eigenvalues. It reports 19.7% and 48.3% fewer samples than jumping to maximal batch on 2D and 40D saddles. It also reports using 10% of the samples of the CABS rule (a published adaptive batch-size heuristic) on a convex quadratic. The question there is whether adding the discrete refinement changes the optimal schedule at large $\eta$, where their SDE should be least accurate.

Why not just…?

Why not just use the gradient noise scale or a signal-to-noise test? This is the real threat. Our formula reduces to a noise-scale quantity when $\eta p h \ll 1$. If most of the floor lives in low-curvature directions, the refinement adds nothing. The continuous-time ablation settles this. We will measure how well each predicts the loss gaps observed in branched runs across training. If the ablation predicts those gaps as well as the full method, we will report that the refinement is unnecessary and that the CBS drift must come from elsewhere. Merrill et al.'s finding that static noise-scale proxies fail for Adam is evidence it will not, but it is not proof.

Why not just run branched training once and reuse the warmup schedule? Merrill et al. found the CBS roughly scale-invariant from 1B to 7B, so one profiling pass might transfer. That transfer is unverified across datasets, learning-rate schedules, and optimizers. Each new setting would need fresh branches at roughly 2B tokens per branch. We test the value of a predictor directly. We ask whether the predicted CBS trajectory tracks the measured one when we change the learning-rate schedule (cosine versus warmup-stable-decay, a schedule that holds the learning rate constant and then decays it at the end), where a transferred warmup would be miscalibrated.

Why not just use the closed-form joint scaling-law schedule? It needs no online statistics and already shows gains. However, it models noise as uncorrected additive variance on a convex proxy, and it reports vanishing benefit under pure FP16. A schedule fixed in advance cannot react to curvature changes such as sharpening toward the edge of stability. If it matches our online rule at equal compute on FineWeb-Edu, the practical case for our method reduces to explanation rather than performance, and we would say so.

Proposed Experiments

We begin on the synthetic quadratics and saddles of Zhao et al. Here the exact Lyapunov solution is computable, so we can check the claimed error ordering directly as $\eta$ sweeps toward $2/\lambda_{\max}$ (the stability limit, with $\lambda_{\max}$ the largest curvature). The expected ordering is that the discrete refined floor stays accurate while the continuous one diverges.

Next we use the airbench94 CIFAR-10 recipe (94% on CIFAR-10 in 3.29 Seconds), which reaches 94.01% in 3.29 seconds and has low run-to-run variance. At that speed we can run exhaustive branches at every checkpoint. The metric is the rank correlation and absolute error between predicted and measured loss gaps across batch sizes.

The central test is language-model pretraining on FineWeb-Edu at 150M and 1B parameters. First, we compare the predicted CBS trajectory with the one measured by the branched protocol. Second, we run the online doubling rule against all baselines. We report final validation loss at a fixed token budget, gradient steps, and wall-clock time including the cost of our curvature estimates. That estimation cost must stay under a few percent of step time, or the gains are illusory.

The decisive checks are two. First, the full method must predict branch outcomes measurably better than its continuous-time ablation, most visibly late in training when sharp directions sit near stability. Second, the online rule must match Merrill et al.'s warmup loss without their branch compute. If only the first holds, we still get a theoretical account of the dynamic critical batch size. If neither holds, the refined mean field view of SGD does not earn its cost.