Training a neural network in low precision creates a specific problem. Each weight update is tiny. When it is smaller than the gap between neighboring representable numbers, ordinary rounding snaps it back to zero and learning stalls. Stochastic rounding fixes this by rounding up or down at random, with probabilities chosen so the result is correct on average. The update then survives in expectation, and large models can train in BF16 with no 32-bit "master copy" of the weights. The catch is that each rounding consumes random bits, and random bits are expensive in silicon. The cheap shortcuts that use only one to four random bits are biased, and that bias makes training blow up.
We propose a narrow bet. Error feedback means recording the rounding error after each update and folding it back into the optimizer's momentum. Master-weight-free optimizers already do this to control noise. We claim it also absorbs the systematic bias of cheap few-bit stochastic rounding. If so, the optimizer update, which is the most memory-bound step in training, can be written as a fused kernel that uses one or two random bits and two or three instructions per element. That instruction and entropy budget is small enough for the simple processing units that sit next to memory banks in processing-in-memory hardware. This matters because processing-in-memory systems are so far used almost only for inference. We test the bet on real GPUs with bit-exact kernels. We also measure how far the usual floating-point emulation of stochastic rounding is from what the kernel actually computes, because that emulation is how nearly every stochastic-rounding result is produced.
Core Claim
Three recent results frame the claim. Stochastic Rounding for LLM Training: Theory and Practice puts stochastic rounding inside the AdamW update and drops 32-bit master weights up to 6.7B parameters. ECO: Quantized Training without Full-Precision Master Weights injects the requantization error into the momentum buffer and reports it works best with stochastic rounding. On Stochastic Rounding with Few Random Bits shows the cheapest few-bit scheme (noise addition, "SRFF") has an exact bias of $-2^{-(N+1)}$ for $N$ random bits, and that training diverges as a result. It also shows the offset-corrected scheme ("SRF") keeps a smaller bias that depends on the input grid. Only a two-stage scheme ("SRC") is unbiased, and it costs extra instructions.
These papers were studied separately. The first two assume full-quality randomness. The third studies few bits without error feedback. The link we propose is simple. Let $w_t$ be the stored low-precision weight, $u_t$ the optimizer update, and $Q$ the rounding operator. The realized rounding error is $e_t = Q(w_t + u_t) - (w_t + u_t)$.
Without feedback, a constant rounding bias $b = \mathbb{E}[e_t]$ adds up linearly, so after $T$ steps the weights drift by roughly $Tb$. That is the divergence seen for SRFF. With ECO-style feedback, $e_t$ is subtracted through the momentum on the next steps. The bias then enters the weights only as a bounded offset. Roughly, it is bias times the momentum's time constant $1/(1-\beta)$, where $\beta$ is the momentum coefficient; this follows from the geometric decay of the feedback term. It is no longer a term that grows with $T$.
The prediction is falsifiable. Under error feedback, SRF with $N = 1$ or $2$ random bits should match full-randomness stochastic rounding in final loss. Without feedback, the same scheme should plateau or diverge. If this holds, the extra correction stage in SRC is unnecessary exactly where it is most costly.
Research Program
The first component is a family of fused optimizer-update kernels written in Triton. Each kernel reads the gradient, the low-precision weight, and the optimizer state once. It computes the AdamW update, rounds with a selectable scheme, and writes back once. The selectable schemes are round-to-nearest, SRFF, SRF, SRC, and full 23-bit stochastic rounding, each with or without ECO feedback. Nothing intermediate goes to global memory. We build on the fused-optimizer design of FlashOptim, which fuses companded 8-bit states and a deterministic 8-bit correction to the weight's last digit. We replace that stored deterministic correction with stochastic rounding plus feedback. Pipelining lessons come from LiquidGEMM, which reports that more than about five instructions per element erases 4-bit gains. We use that roughly five-instruction figure as our budget. Our kernel is elementwise rather than a matrix multiply, so the budget acts as a design target, not a proven limit.
The second component is the random-bit source, which is where emulation and hardware differ most. Emulated training calls a high-quality generator per element. Hardware would use a small counter-based hash or a short linear-feedback shift register, whose bits are correlated across neighboring parameters and steps. Correlation does not bias the expectation. It can, however, correlate rounding errors over time and weaken the temporal averaging that stochastic rounding relies on. We parametrize the source by bits per element and by generator family. Following the finding in AdamW-SR that seeds must be shared across data-parallel ranks, the generator state is shared across ranks.
The third component is a measurement protocol for the gap between simulation and silicon. We train each configuration twice, once with floating-point emulation of the rounding and once with the bit-exact fused kernel. We report how far the two loss trajectories drift apart. We also build a simple instruction-and-bytes cost model of each kernel. We fit it against measured GPU time and report its prediction error. Only then do we use the same model to project cost on a near-memory processing unit, with that measured error attached as an error bar.
We are explicit about scope. We have no processing-in-memory silicon. The in-memory results are projections, labeled as such. Their credibility comes from how well the same model predicts hardware we can measure.
Processing-in-memory is the regime this work targets, not the mechanism. AQPIM shows digital in-memory processing elements can run quantization and table lookups inside DRAM banks, but only for inference. The elementwise optimizer update has no data reuse, which makes it the natural training workload to move into such units, provided its per-element cost is tiny.
We deliberately keep analog crossbars out of the core claim. Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators finds that noise at the array output is far more damaging than weight noise. Dynamic Symmetric Point Tracking shows analog updates also carry device-specific asymmetry. Neither issue arises in exact digital near-memory arithmetic. Using analog noise as a free source of randomness is tempting, but it would need that noise to be zero-mean and stable, which both papers contradict.
Baselines
The naive reference is standard mixed precision: BF16 compute with FP32 master weights and FP32 Adam states. Every other configuration is judged relative to its final validation loss and per-parameter memory. Round-to-nearest into BF16 with no master weights and no feedback serves as the stagnation floor.
The strongest existing methods are two master-weight-free optimizers run with full-quality randomness. The first is AdamW-SR from Stochastic Rounding for LLM Training: Theory and Practice. It reports a 2.6% perplexity improvement over mixed precision on GPT-2 350M and up to 1.54x throughput on 6.7B models. The second is ECO with full stochastic rounding from ECO. It reports a drop from 12 to 9 bytes per parameter on a sparse mixture-of-experts model while matching the FP32-master-weight validation loss. ECO with full randomness is the one to beat, because it already contains our feedback ingredient. Our method differs only in how cheap the randomness is.
The deterministic alternative is FlashOptim's split-weight scheme, which reaches 7 bytes per parameter. We run it both at its native memory and at an iso-memory point, where our saved correction byte is given back to it.
SRC from On Stochastic Rounding with Few Random Bits, at the same $N$, is the unbiased few-bit reference.
The key ablation crosses {round-to-nearest, SRFF, SRF, SRC, full stochastic rounding} with {no feedback, ECO feedback}, at $N \in \{1, 2, 4, 8\}$. This isolates whether feedback, and not the rounding scheme alone, rescues the cheap schemes.
To keep the comparison fair, all methods share the same data order, token budget, model, and fused-kernel scaffolding. Each method gets the same learning-rate sweep of five values, because AdamW-SR reports that stochastic rounding prefers higher learning rates. We report three seeds.
We count our method as winning if one condition holds. SRF with $N \le 2$ plus feedback must reach final loss within seed-to-seed spread of ECO with full randomness. It must also use fewer instructions and random bits per element than SRC, at equal or lower memory than FlashOptim.
Why not just…?
Why not just keep FP32 master weights? It is the safe default. But it is exactly what makes in-situ training impractical near memory, where capacity is the binding limit. FlashOptim's accounting puts standard AdamW at 16 bytes per parameter. The master-weight-free results above already show removal costs nothing in quality when randomness is good. Our question is only what the randomness must cost, so mixed precision serves as the reference point rather than a competitor to the claim.
Why not just use SRC, which is already unbiased? This is the real threat. If SRC fits inside the fused kernel's instruction budget with no measurable slowdown, the GPU case for our method collapses. The contribution would then shrink to a smaller entropy requirement for near-memory units. Our kernel measurements settle this directly: we report SRC and SRF instruction counts and measured throughput side by side. If they match on GPU, we say so and rest the case only on the projected near-memory budget.
Why not just use round-to-nearest with error feedback and skip randomness entirely? ECO reports an elevated noise floor under round-to-nearest. Training with Fewer Bits reports that larger batches rescue stochastic rounding but not round-to-nearest. The zero-bit column of our ablation tests whether any random bits are needed at all. If round-to-nearest plus feedback matches, that is a stronger and simpler result, and we would report it as the finding.
Why not just store a deterministic correction byte, as FlashOptim does? It needs no random bits. The iso-memory comparison decides it. FlashOptim spends one extra byte per parameter on the correction. If our zero-extra-byte configuration reaches the same loss, the byte is saved. If it does not, deterministic storage remains the better trade.
Proposed Experiments
The sweep follows prior settings so that numbers are comparable. It starts with nanoGPT on Shakespeare and GPT-2 350M, matching the few-random-bits paper. It then moves to 30M to 800M-parameter Transformers on C4, matching ECO, and to GPT-2 on FineWeb10B, matching FlashOptim. The final setting is Llama-3.2-3B fine-tuning on GSM8K, where Training with Fewer Bits studied stochastic rounding. No reported GSM8K score for that setting was found on Lacuna, so we will establish our own mixed-precision reference.
The metrics are validation loss or perplexity, task accuracy, bytes per parameter, measured kernel throughput, instructions per element, and random bits per element.
The decisive checks are four.
First, without feedback, SRFF should diverge at the thousands-of-steps horizon reported by Fitzgibbon and Felix. With feedback, it should instead stall at a bounded loss gap. SRF with feedback at $N = 2$ should close that gap.
Second, sweeping the momentum coefficient $\beta$ should change the residual gap of biased schemes roughly in proportion to $1/(1-\beta)$. That dependence would confirm the proposed mechanism rather than an accidental fit.
Third, emulated and bit-exact runs should agree within seed spread for full randomness. We expect them to diverge measurably for linear-feedback-shift-register sources with few bits. Such a divergence would quantify how much published emulated results can be trusted.
Fourth, the cost model's prediction error on GPU kernels must fall below 20%. Otherwise we will not report near-memory projections at all.
References
- Stochastic Rounding for LLM Training: Theory and Practice
- ECO: Quantized Training without Full-Precision Master Weights
- On Stochastic Rounding with Few Random Bits
- FlashOptim: Optimizers for Memory-Efficient Training
- LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
- AQPIM: Breaking the PIM Capacity Wall for LLMs with in-Memory Activation Quantization
- Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators
- Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training
- Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding