Training with larger mini-batches lets a cluster finish an epoch faster. The standard recipe, the linear scaling rule, multiplies the learning rate by $k$ whenever the batch grows by $k$. The rule eventually breaks: ResNet-50 on ImageNet keeps its accuracy up to a batch of roughly 8,192 and then degrades sharply. We propose that a measurable part of this ceiling comes from evaluating the last iterate rather than an average of recent iterates. When the scaled step size approaches the stability limit of the sharpest directions of the loss, the parameters bounce back and forth across the valley. That bouncing inflates the last iterate's error, and averaging cancels it almost exactly. The obstacle is that any average also drags in stale, early parameters, which is the classical problem of initial iterate bias. So the right tool is an average whose window is measured in epochs and which subtracts the lag it would otherwise carry.
In two sentences: under linear scaling, the last iterate's noise grows as the step size nears the edge of stability, while the noise of a fixed-epoch average stays the same at every batch size. A bias-corrected average therefore lets the batch grow until training truly diverges, rather than until accuracy starts to slip. If the ceiling instead comes from the optimizer landing in a different, worse basin, averaging will not move it, and our experiments are built to tell the two cases apart.
Core Claim
Consider the simplest model in which this can be computed exactly: stochastic gradient descent (SGD) on a quadratic loss. Take one curvature direction with eigenvalue $\lambda$, per-example gradient noise variance $\sigma^2$, batch size $B$ and step size $\eta$. The error obeys $e_{t+1} = (1-\eta\lambda)e_t + \eta\xi_t$, where the noise $\xi_t$ has variance $\sigma^2/B$. The stationary variance of the last iterate is
$$\mathrm{Var}(e_t) = \frac{\eta\,\sigma^2}{B\,\lambda\,(2-\eta\lambda)},$$
which comes from summing the geometric series of squared contraction factors. Under linear scaling, with $\eta = k\eta_0$ and $B = kB_0$, the ratio $\eta/B$ is preserved. This invariance is the "stochastic temperature" argument of Jastrzebski et al.. It leaves one leftover factor, $1/(2-k\eta_0\lambda)$, and that factor blows up in the sharpest directions as $k$ grows.
Now average the last $L$ steps, with the window held fixed in epochs so that $L = L_0/k$. Once the window is longer than the correlation time, the variance of the average is the long-run variance of this autoregressive process divided by $L$:
$$\mathrm{Var}(\bar e) \approx \frac{\sigma^2}{B\,\lambda^2\,L} = \frac{\sigma^2}{B_0\,\lambda^2\,L_0}.$$
This quantity depends on neither $k$ nor $\eta$. Iterate averaging therefore removes exactly the discrete-time amplification that linear scaling cannot. Bias tells a similar story. In flat directions, where $k\eta_0\lambda \ll 1$, forgetting of the initialization per epoch is unchanged by $k$. In sharp directions it is faster, because $\log(1-x)\le k\log(1-x/k)$, until $k\eta_0\lambda$ approaches 2.
The price of averaging is lag. A window of $L_0$ epochs makes the estimate trail the current iterate in the flat directions, where the model is still moving. That lag is the initial-iterate-bias problem in a new setting, and it determines how long the window can usefully be. We claim that a bias-corrected average lets the window be long enough to cancel the amplification without paying the lag. The correction we have in mind is the displacement-corrected exponential moving average (BEMA) of EMA Without the Lag, re-parameterized in epochs rather than steps. The predicted result is a batch-size ceiling set by outright divergence ($k\eta_0\lambda_{\max}\approx 2$) rather than by gradual degradation.
Relation to Prior Work
Two papers motivate the gap from opposite sides. How to Scale Your EMA derives the rule $\hat\rho=\rho^\kappa$ for scaling an exponential moving average's decay with batch size. Its derivation uses a small-learning-rate stochastic differential equation, so it cannot see the $1/(2-\eta\lambda)$ term, and it adds no correction for lag. We keep its epoch-invariant window, analyze the large-step regime it excludes, and add debiasing. Accurate, Large Minibatch SGD established linear scaling with gradual warmup and documented the ceiling, but evaluated only the last iterate. We ask how much of its elbow is a property of the estimator rather than the trajectory.
Three papers supply the method and constrain the design. Understanding SGD with EMA proves per-eigendirection exponential bias decay for an exponential moving average, but only at batch size 1. Parallelizing SGD for Least Squares and Beating SGD Saturation analyze mini-batching combined with tail averaging (averaging only a final window of iterates) and show that tail averaging permits aggressive constant step sizes. None of the three holds the epoch budget fixed while scaling $\eta\propto B$, which is the regime where the amplification factor appears.
One paper is the closest competitor. SWAP averages independently refined workers after a large-batch phase. It closes the CIFAR-100 gap, but it requires a hand-picked switch epoch and parallel small-batch runs. Our proposal averages a single large-batch trajectory with no switch. It stays at large batch for the entire run, which SWAP does not.
Research Program
The theory part extends the quadratic calculation above to the full spectrum and to fixed-epoch budgets. It derives the excess risk of three estimators under $(\eta,B)=(k\eta_0,kB_0)$: the last iterate, a tail average, and BEMA with epoch-parameterized coefficients. It also yields, for each estimator, the critical $k$ at which risk exceeds the $k=1$ risk by a fixed tolerance. We will extend the result to the mis-specified, heteroscedastic noise model of Jain et al., where stable step sizes are already noise-limited.
The method part defines batch-aware BEMA. Its coefficients $\alpha_t,\beta_t$ are functions of elapsed epochs rather than steps, and its decay is set by $\rho^\kappa$. One design choice speaks directly to the warmup tension: the anchor that BEMA subtracts can be the random initialization $\theta_0$ or the iterate at the end of warmup. Warmup iterates move little, so anchoring at $\theta_0$ may over-correct. Batch-normalization statistics are recomputed for every averaged model, as SWAP does.
The diagnostic part tests whether the mechanism is present in real networks. We estimate the top Hessian eigenvalue by power iteration during training and measure the sign-alternation of parameter displacement along that eigenvector. The theory predicts that the last-iterate versus average gap tracks $k\eta_0\lambda_{\max}$ and the oscillation rate.
Baselines
The naive reference is the recipe of Accurate, Large Minibatch SGD: linear scaling, five epochs of warmup, step decay, and evaluation of the last iterate. Its reported ResNet-50 ImageNet result is 23.74% top-1 error at batch 8,192 versus 23.60% at batch 256, with error rising sharply beyond 8,192. We rerun it at batch sizes 8k, 16k and 32k under identical code.
The strongest decay-based competitor is the same pipeline with the learning-rate schedule (cosine or linear-to-zero, plus its endpoint) tuned separately at each batch size. Annealing also suppresses last-iterate noise, so this is the real threat. It receives the same number of tuning trials as our method.
The strongest averaging competitor is a plain exponential moving average scaled by $\rho^\kappa$, following How to Scale Your EMA. That paper reports no numeric CIFAR or ImageNet supervised-accuracy value in our file; it reports BYOL parity at batch 24,576 with the batch-4,096 baseline. Comparing against this baseline isolates our key ingredient, the lag correction. A tail average beginning after warmup with an epoch-fixed window provides a second averaging-without-correction ablation.
SWAP is the reference for large-batch generalization recovery. It reports 78.18% on CIFAR-100 versus 75.84% for plain large-batch training and 77.01% for small-batch training (SWAP). We run it with the same total epochs and its switch epoch tuned on the same budget.
All methods share data order, epochs, warmup length and tuning trials. Averaging adds zero extra gradient computations. Beating a baseline means one of two things. The first is a larger critical batch size, defined as the largest batch whose accuracy stays within 0.3 points of the small-batch reference. The second is better accuracy at a batch size where the baselines have already degraded.
Why not just…?
Why not just anneal the learning rate to zero at the end? In the quadratic model, annealing also cancels the $1/(2-\eta\lambda)$ term. Under a fixed epoch budget with few total steps, however, the shrinking step size also halts progress in the flat directions, where initialization is still being forgotten. Averaging cancels the noise without stopping that motion. Which effect dominates at large $k$ is an empirical question, and this baseline gets a full per-batch tuning budget. If tuned annealing matches batch-aware BEMA at 16k and 32k, our claim reduces to "schedules suffice," and we will report it as such.
Why not just use a $\rho^\kappa$ exponential moving average with no lag correction? This is the most likely way the idea fails. If the epoch-fixed window can be short enough that lag is negligible while still cancelling the amplification, BEMA adds nothing. The ablation decides it. Debiasing earns its place only if the best uncorrected window either leaves amplified noise in place or trails in flat directions measurably, which should show up as a lower synthetic risk and as accuracy differences that grow with $k$. The evidence that lag matters elsewhere comes from fine-tuning, where BEMA reached a GSM8K plateau in about half the steps (EMA Without the Lag). That does not by itself transfer to this setting.
Why not just use SWAP? SWAP returns to small batches for its refinement phase and depends on a switch epoch whose misplacement causes divergence or lost efficiency. If it beats single-trajectory averaging at equal epochs, that tells us something important: the large-batch gap is a basin-selection effect that temporal averaging cannot repair, consistent with the flat-minima account of Jastrzebski et al..
Proposed Experiments
The first experiment is synthetic. We use overparameterized linear regression with eigenvalues $\lambda_i=i^{-2}$, $d=2000$, Gaussian features, a setting matching Understanding SGD with EMA. We add a heteroscedastic variant and the $d=1000$ non-parametric ensemble of Beating SGD Saturation. We sweep $k$ from 1 to 1,024 at a fixed sample budget and decompose risk per eigendirection into bias and variance. The decisive check is whether last-iterate risk follows the $1/(2-k\eta_0\lambda_{\max})$ curve while averaged risk stays flat until divergence. No reported result was found on Lacuna for fixed-epoch linear-scaling sweeps in this setting, so all numbers here are new.
The second experiment uses ResNet-18 on CIFAR-10 and CIFAR-100 with batches from 128 to 16k, fixed epochs, and linear scaling with warmup. We locate the critical batch size for each estimator and log the top Hessian eigenvalue and the oscillation rate along its eigenvector. The decisive check combines two signals: whether the critical batch size moves upward, and whether the last-iterate versus average gap correlates with $k\eta_0\lambda_{\max}$. A gap that is flat in $k$ would falsify the mechanism.
The third experiment is ResNet-50 on ImageNet-1k at 8k, 16k and 32k for 90 epochs, compared against the 23.74% and 23.60% references. The decisive check is whether batch-aware BEMA holds within 0.3 points of the batch-256 error at 16k, where the last iterate is known to degrade. A final ablation compares anchoring at $\theta_0$ versus at the end of warmup, and tests whether a shortened or removed warmup becomes tolerable once the average excludes early iterates.
References
- Finding Flatter Minima with SGD
- EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
- How to Scale Your EMA
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression
- Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification.
- Beating SGD Saturation with Tail-Averaging and Minibatching
- Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well