Adam-style optimizers keep two running averages per parameter: a momentum and a running mean of squared gradients. These averages take about four times the memory of the weights. The main fix in practice is to store them in 8 or 4 bits, re-quantizing after every update. Each stored state is then no longer a floating-point running average. It is a small random dynamical system: shrink by β, add a fresh gradient term, round back onto a coarse grid whose scale is set by the largest value in the block. Nobody has asked what distribution this rounded recursion settles into, or how that distribution differs from the one the full-precision optimizer would reach. The existing 4-bit recipes (small blocks, row/column normalization, grids that exclude zero) are fixes discovered by trial. They come with no account of when they will stop working at 3 or 2 bits, at larger β, or in longer runs.
We propose to compute that settled distribution with population dynamics. Population dynamics is the statistical-physics technique that solves "the distribution must reproduce itself under this random update" equations by evolving a large pool of sample values. Each block of a quantized optimizer is exactly such an equation: a population of state values under a shared, block-maximum-dependent rounding. Solving it from cheaply logged gradient statistics gives a predictive map from tail heaviness, block size, bit width and β to optimizer health. The same tool can then choose, per layer, a grid, a rounding rule and an effective decay that make the rounded recursion settle where the full-precision one would. The core bet is that moving below 4 bits is a fixed-point-matching problem, not a reconstruction-error problem.
Core Claim
Write $s_t$ for one stored state coordinate, $x_t$ for its fresh input ($g_t$ for momentum, $g_t^2$ for the second moment), and $Q_S$ for rounding onto a grid scaled by the block statistic $S$, for example the absolute maximum over the $B$ coordinates of the block. A quantized optimizer then runs
$$s_{t+1} = Q_{S_t}\big(\beta s_t + (1-\beta) x_t\big), \qquad S_t = \max_{j \in \text{block}} |\beta s^{(j)}_t + (1-\beta) x^{(j)}_t|.$$
The stationary law of $s$ solves a distributional fixed-point equation whose interaction kernel takes $B$ incoming values. This is the structure of the cavity equations in Gibbs states and the set of solutions of random constraint satisfaction problems. It has two failure modes that a fixed-point analysis exposes and a per-step error analysis misses.
The first failure mode is stalling. With round-to-nearest, any increment $(1-\beta)x_t$ smaller than half the grid spacing $\Delta$ is erased, so the state freezes. For $\beta_2 = 0.999$ this happens unless the gradient is about a thousand grid steps away.
The second failure mode is noise amplification. With unbiased stochastic rounding, each step injects variance up to $\Delta^2/4$. The recursion keeps a fraction β of that noise, so the settled quantization noise has variance about
$$\mathrm{Var}_{\text{quant}} \approx \frac{\Delta^2}{4(1-\beta^2)},$$
which follows from summing the geometric series of retained noise. That is roughly $125\Delta^2$ at β = 0.999, enough to swamp a 4-bit state, and only about $5\Delta^2$ for momentum at β = 0.9.
Heavy-tailed gradients make the block maximum grow with block size. This pushes typical coordinates toward the zero bucket, which is the "zero-point problem" reported in Memory Efficient Optimizers with 4-bit States. Our claim is that a cheap population-dynamics solve of this equation will quantitatively predict where these failures appear in real training. It will also produce per-layer quantizer settings that let 3-bit states match the trajectories of 32-bit AdamW where the current 4-bit recipe, pushed to 3 bits, does not.
Where This Sits in Prior Work
Supply the problem and the incumbent. Two papers built low-bit optimizer states. 8-bit Optimizers via Block-wise Quantization introduced block-wise dynamic quantization of states. It also reported that 8-bit AdaGrad failed without stochastic rounding because its accumulated range kept widening, which is a fixed-point failure in our terms. Memory Efficient Optimizers with 4-bit States diagnosed the zero-point and outlier problems and fixed them with 128-element blocks, rank-1 normalization and a zero-excluding grid. Both papers judge quantizers by how well they reconstruct a single snapshot. We judge them by the distribution the rounded recursion converges to, and we treat their design choices as points in a space our solver can search.
Supply the method. Classical population dynamics is the fixed-point solver used across these works. Thermodynamics of the Lévy spin glass adapted population dynamics to couplings with divergent second moments. It split a sparse backbone of strong bonds from a weak background with an ε-cutoff, and found that heavier tails let a few strong bonds dominate local fields. We reuse that strong/weak split. Blocks whose maxima are set by a few outlier coordinates are the optimizer analogue of a strong-bond backbone. The prediction to test is that the block maximum grows like $B^{1/\alpha}$, where α is the gradient tail exponent, and that this growth sets the rate at which typical coordinates fall to zero. State following, which tracks a metastable state with coupled populations as parameters drift, supplies the technique for re-solving the fixed point as gradient statistics shift during training, rather than assuming one stationary regime.
Motivate a lens but do not supply an algorithm. Disordered Dynamics in High Dimensions uses cavity and dynamical mean-field arguments to reduce gradient descent to a single-coordinate stochastic process. It does so only for idealized models and never for stored optimizer states. We take the single-coordinate reduction seriously but apply it to the optimizer's own memory, where the reduction is closer to exact because the coordinates of a block interact only through the shared scale.
Constrain the design. Deconstructing What Makes a Good Optimizer for Language Models finds that per-parameter adaptivity matters mainly at the embedding/logit layers and LayerNorm, and that inner layers tolerate layer-wise scalars. Adafactor factors the second moment into row and column sums. Together they mean that aggressive second-moment quantization of inner layers may be unnecessary. The real stakes are in momentum, which AdaMeM shows cannot be dropped at large batch, and in the few layers that need fine-grained adaptivity. We design the study around this constraint rather than against it.
Research Program
The work has three parts.
Measurement. We instrument training of LLaMA-style models from 60M to 350M parameters on C4, following the GaLore setup. Every few hundred steps, for every block of every layer, we log summary statistics of the gradient input: a tail-exponent estimate, quantiles, and the lag-one autocorrelation. Raw gradients are not stored.
The solver. For a given layer and quantizer configuration, the solver maintains a population of $M$ state values. Each update has three steps. Selection draws $B$ population members and $B$ inputs from the logged input law. Interaction applies one rounded update with the block-maximum scale. Replacement puts the results back into the population. A parallel population with no rounding gives the full-precision reference law. The solver reports four summaries of the gap: the fraction of states stuck at zero, the fraction stalled (unchanged for many steps), the settled quantization noise, and a distance between the rounded and reference laws of the preconditioner $1/\sqrt{v}$. That last quantity is what actually scales Adam's step.
Two simplifications need checking. The solver treats inputs as independent across steps, and the logged autocorrelation lets us test whether that matters. It also ignores feedback from the state to the gradient, which we bound empirically by comparing predictions against real runs.
Calibration. For each layer, we search a small design space and pick the cheapest setting whose solved fixed point lies within a tolerance of the reference. The design space covers bit width (2–4), block size, grid shape (linear, dynamic-exponent, or quantile-of-solved-law), and rounding rule (nearest, stochastic, or a hybrid that rounds stochastically only when the increment is below half a grid step). It also includes a reduced effective β for the stored state, which trades statistical averaging against noise amplification. Solving costs seconds per layer on CPU, so calibration can be repeated as gradient statistics drift, using state following.
Baselines
The naive reference point is full 32-bit AdamW. Every quantized method is compared to its validation loss curve on the same data, token budget and learning-rate schedule. A method "matches" it if the final C4 validation perplexity is within a pre-registered tolerance, set from the seed-to-seed spread of three AdamW seeds.
The strongest existing method is the 4-bit AdamW of Memory Efficient Optimizers with 4-bit States, run with its published block size, rank-1 normalization and zero-excluding grid. The paper reports total training memory for LLaMA-7B dropping from 75.40 GB to 31.88 GB on Alpaca with a loss curve matching full precision. On GLUE with RoBERTa-Large it reports results comparable to 32-bit, but the file gives no single score. We run it at its native 4 bits and also at 3 bits with the same recipe. Beating it means one of two things: matching full precision at 3 bits where its 3-bit version does not, or matching it at 4 bits with fewer per-layer special cases. It is strongest because it is the only published recipe that already handles the zero-point problem at low bit width.
We also run the 8-bit block-wise optimizer of 8-bit Optimizers via Block-wise Quantization as the robust middle ground. On the memory-elimination side, we run Adafactor, which reports 25.9 BLEU versus Adam's 25.6 on WMT'14, and AdaMeM as the momentum-preserving factored alternative. AdaMeM has no reported C4 perplexity number on Lacuna. On the projection side, we run GaLore, which reports a GLUE average of 85.89 at rank 4. GaLore is compared at equal optimizer-state bytes, not equal bits.
All methods share data, token count and seeds, and each gets the same learning-rate sweep of five values.
Two ablations isolate the key ingredient. The first replaces the solved fixed-point law with the empirical histogram of the current full-precision state, giving a quantile grid fitted to a snapshot. The second keeps the solver but removes the block-maximum coupling ($B = 1$). If either ablation matches the full method, the fixed-point view is not doing the work.
Why not just…?
Why not just use the existing 4-bit recipe, which already works? At 4 bits it may well be enough, and we expect our method to roughly tie it there. The open question is whether that success extends. The 8-bit AdaGrad failure in 8-bit Optimizers via Block-wise Quantization shows that recursions with slow forgetting break in ways a snapshot analysis does not foresee. The 3-bit runs of the incumbent recipe, and runs at higher $\beta_2$, are the experiment that settles whether there is anything left to gain.
Why not just fit the quantization grid to the observed histogram of the states? This is the most serious threat. It is cheap and captures most of the marginal shape. It cannot see the dynamical bias, though: a grid fitted to today's full-precision states says nothing about whether the rounded recursion will stall, or drift to zero, after thousands of steps. The snapshot-quantile ablation in Baselines tests this directly. If it matches the solver-calibrated quantizer at 3 bits, we will report the solver as a diagnostic tool only and drop the design claim.
Why not just simulate the scalar recursion by brute force, or factor the second moment away? Brute-force Monte Carlo of the rounded recursion is what population dynamics is here. The contribution is the block-coupled fixed-point formulation and the tail-aware analysis of the block maximum, not the name. Factoring the second moment, following Adafactor and Deconstructing What Makes a Good Optimizer, is a genuine competitor for inner layers. For that reason, our headline target is quantized momentum together with quantized second moments only in the layers that need per-coordinate adaptivity. We compare against AdaMeM at matched memory.
Proposed Experiments and Decisive Checks
Prediction test. Before any calibration, we sweep bit width, block size and β on 60M and 125M C4 models. For each configuration we record the measured zero fraction, stall fraction and preconditioner error of the real quantized optimizer, and correlate them with the solver's predictions from logged statistics. The prediction claim fails if the solver ranks configurations no better than the snapshot-histogram heuristic. The claim survives if the solver places the failure boundary, meaning the bit width at which perplexity departs from AdamW, within one bit and one block-size octave across layers. A sharper sub-test checks the predicted $B^{1/\alpha}$ growth of zero-bucket mass against measured α per layer.
Design test. We train 125M and 350M models on C4 and fine-tune RoBERTa-Large on GLUE. We compare full precision, the incumbent recipe at 4 and 3 bits, the two ablations, and the solver-calibrated quantizer at 3 bits, plus a 2-bit momentum variant. If the 350M results hold, we finish with LLaMA-7B on Alpaca to measure total memory against the reported 31.88 GB.
Large-batch stress test. Momentum fidelity matters most at large batch, where AdaMeM shows that removing it hurts. We therefore repeat the 125M runs at four times the batch size and test whether low-bit momentum preserves the large-batch advantage that full momentum provides.
Decision rule. The project succeeds if the calibrated 3-bit quantizer matches AdamW within seed noise at 350M while the incumbent recipe at 3 bits and the snapshot-quantile ablation do not.
References
- Gibbs states and the set of solutions of random constraint satisfaction problems
- Memory Efficient Optimizers with 4-bit States
- 8-bit Optimizers via Block-wise Quantization
- Thermodynamics of the Lévy spin glass
- Following Gibbs states adiabatically: the energy landscape of mean field glassy systems
- Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning
- Deconstructing What Makes a Good Optimizer for Language Models
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- AdaMeM: Memory Efficient Momentum for Adafactor
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection