Grading Online Batch-Size Controllers Against Measured Critical Batch Size

Many training algorithms change the batch size (the number of examples averaged into each gradient step) as training goes on. Each one comes with a convergence proof, and each one is usually tested only by showing that it reaches a good final loss on a small model. None has been checked against the quantity it is supposed to track: the critical batch size, the largest batch beyond which adding more examples per step stops buying proportionally fewer steps. That quantity can now be measured directly. Critical Batch Size Revisited (also indexed as Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training) forks short training runs at several batch sizes from saved checkpoints and reads off where larger batches stop paying. With this "branched training" they found that the critical batch size rises early in language-model pre-training, then plateaus, and follows the same trajectory at 1B and 7B parameters. They also found that the classical theoretical proxy, the gradient noise scale, fails to predict it under the Adam optimizer.

We propose to use branched-training measurements as ground truth and score every online batch-size rule against them. The rules include gradient-statistics tests, similarity heuristics and loss-based probes. We then ask a sharper question that existing papers leave open. Does any online rule beat simply profiling once and replaying the schedule? We expect the answer to be "yes, but only when the training distribution or schedule shifts mid-run," and we design a stress test to settle it.

Core Claim

Our hypothesis has two parts. First, online rules built on gradient statistics drift away from the measured critical batch size under Adam, while outcome-based probes stay close to it. The gradient-statistics rules include the norm and inner-product tests used by AdAdaGrad, AdaBatchGrad and the Local-SGD variant of Lau et al.. An outcome-based probe takes one or a few steps at batch $B$ and at batch $2B$ from the same state and compares the loss reduction per token on held-out data. The suspected reason is the same one Merrill et al. give for the noise scale: Adam rescales each coordinate, so it breaks the relation between raw gradient variance and the usefulness of a larger batch. A probe that measures outcomes directly does not depend on that relation.

Second, a probe-driven controller should match a replayed profiled schedule on stationary data. It should beat that schedule when the data mixture changes mid-training, because a fixed schedule cannot see the change.

The new element is not another adaptive rule. It is a measurement protocol that grades rules against an independently measured target, together with a controller whose design is chosen by that protocol. The controller is an online, low-cost version of Merrill's branch comparison. It is also a non-convex, Adam-compatible descendant of the two-track loss comparison in Batch-Expansion Training, which was only ever proven and tested on strongly convex linear models.

Research Program

The first step is building the oracle. We train decoder-only Transformers of roughly 150M and 400M parameters with Adam on C4 and on FineWeb. Merrill et al. report scale invariance between 1B and 7B, so smaller models are a defensible proxy, and we check the claim between our two sizes. At about twenty checkpoints we run Merrill-style branches at batch sizes spaced by factors of two, scaling the learning rate with the square root of the batch size as they do. We record the largest batch whose branch loss matches the smallest-batch branch at equal tokens, within a fixed tolerance. This gives a measured critical-batch-size trajectory, with uncertainty taken from repeated branches under different data orders.

The second step is grading the candidate signals. We log each candidate signal along a single reference run and convert it into an implied batch size. The candidates are:

  • the norm test and inner-product test, computed from per-sample gradients as in AdAdaGrad;
  • the same tests computed cheaply from the two halves of a data-parallel batch;
  • SimiGrad's cosine similarity between worker groups (SimiGrad);
  • the gradient noise scale;
  • a preconditioned norm test, which measures variance after dividing by Adam's second-moment estimate;
  • the outcome probe at one-, four- and sixteen-step lookahead.

The preconditioned norm test is the key diagnostic. If it closes most of the gap, the failure of gradient statistics is specifically about preconditioning, and the cheaper fix wins. For each signal we report how closely its implied batch size follows the oracle over training. We measure this as the rank correlation and as the mean absolute log-ratio between the two. We also report the signal's compute and memory overhead.

The third step is closing the loop. The best-graded signal drives a controller that doubles the batch when the probe says $2B$ is nearly as efficient per token as $B$. Batch sizes are restricted to powers of two, which avoids the rounding problem raised for continuous optimal-control schedules. We run this controller end to end.

As a secondary, smaller check on whether the conclusions transfer beyond language models, we repeat the oracle-and-grading loop for ResNet-18 on CIFAR-10. AdAdaGrad and SimiGrad both report results in that setting.

Baselines

The naive reference is a fixed batch at two settings: a small batch, and the largest batch that fits on the hardware. Both use a tuned cosine learning-rate schedule. Merrill et al. report this comparison for OLMo 1B. Final pre-training loss was 2.6057 for the small-batch control and 2.5962 for the fixed large-batch control, against 2.5891 for batch-size warmup, which used 43% fewer gradient steps than the fixed-batch baseline.

The strongest existing method is Merrill's profiled warmup. We measure the critical-batch-size trajectory once on the 150M model, then replay it as a fixed schedule on the 400M model and on new seeds. This is the hardest competitor because it uses the ground truth itself and pays no online overhead; the extra cost was paid during profiling.

The strongest existing online methods are AdAdaGrad's inner-product test and Lau et al.'s norm test. On C4 with MicroLlama, Lau et al. report validation loss 4.55 against 4.82 for a constant batch. We also run SimiGrad, which reports up to 40.44% wall-clock savings on BERT-Large with 10–15 ms of overhead per measurement.

The ablation replaces the outcome probe with the preconditioned norm test while keeping the same controller. This isolates whether measuring outcomes, rather than simply correcting for Adam's preconditioning, is what matters.

Fairness rules apply to all methods. Every method gets the same token budget and the same learning-rate scaling rule. Each gets an equal tuning budget of eight trials for its own threshold. All are run on the same hardware with the same rounding to powers of two. The profiling cost of the replayed schedule is reported separately, so it is visible rather than hidden.

The proposal counts as winning under two conditions. On stationary data, the controller must match the replayed schedule within seed noise on final loss and on steps, while adding less than 5% wall-clock overhead. Under data shift, it must reach strictly lower loss than the replayed schedule at equal tokens, and beat it on time to a target loss.

Why not just…?

Why not just profile once and replay the schedule, since the critical batch size is scale-invariant? This is the real threat, and on stationary data we expect it to tie or win. We will report that outcome honestly, not argue around it. The decisive test is a mid-run data-mixture switch from FineWeb to FineWeb-Edu, a filtered subset that the FineWeb paper shows changes downstream behavior substantially. We pair it with a mid-run learning-rate rewarm. We re-measure the oracle after the switch. If the trajectory does not move, online control is unnecessary in practice, and that is a useful negative result for the field.

Why not just use a fixed doubling schedule or linear batch warmup? Evidence that fixed switching wastes samples exists only on synthetic quadratics: Zhao et al. report 19.7–48.3% extra samples. The file has no deep-network evidence either way. We therefore include a doubling schedule with its period tuned under the same eight-trial budget, and grade its implied trajectory against the oracle like every other method.

Why not just use the gradient noise scale? Merrill et al. report that it misaligns with measured critical batch size under Adam. We include it to confirm that result at our scales, and to measure how much of the gap the preconditioned variant recovers.

Why not just decay the learning rate on a fixed batch? That is the small- and large-batch reference above. It controls noise but does not reduce the number of sequential steps, and step count is the quantity that matters for parallel training.

Decisive Checks

Three results would decide the project. The first is whether outcome probes achieve clearly higher rank correlation with the oracle than gradient-statistics tests, across both model sizes. The second is whether the preconditioned norm test closes that gap; if it does, we would adopt it as the cheaper method. The third is whether the oracle trajectory moves after the data switch, and whether only the online controller follows it. If gradient-statistics tests already track the oracle well, or if the trajectory does not move under the data switch, the second part of the core claim fails. The deliverable then becomes the grading benchmark itself, together with a recommendation to profile once and replay.