Warm Restarts for Paged Optimizer State Under an On-Chip Memory Budget

Adam, the default optimizer for training neural networks, keeps two extra numbers for every weight. One is a running average of the gradient. The other is a running average of the squared gradient, which sets a separate step size for each weight. On a small accelerator or microcontroller, that extra state is often what stops training from fitting in fast on-chip memory (SRAM). One natural fix is to train only a few blocks of weights at a time. The optimizer state for those blocks stays on chip, and the state for every other block is discarded until its turn comes back. This proposal argues that the fix fails in a predictable way. Each time a block comes back with empty statistics, it repeats Adam's well-known first-step instability: oversized, sign-only updates that cause loss spikes. These are the same cold-start failures that warmup schedules exist to hide.

The core idea is to keep a very small, permanent summary of each block's squared-gradient history on chip. When a block returns, the summary is used to rebuild that block's per-weight step sizes, and each block keeps its own step counter. The question we test is how little persistent state suffices to make a paged optimizer behave like full Adam, while moving almost none of Adam's state across the external memory bus.

Why this gap exists

Two literatures meet here without talking to each other. On the hardware side, most work on fitting networks into on-chip memory compresses or schedules inference parameters and activations. Deep Compression combines pruning, quantization and Huffman coding to fit weights in SRAM. Fpga Based Implementation of Deep Neural Networks Using On-chip Memory Only uses 2–3 bit weights to avoid external memory. Auto WS and layer-centric tile fusion schedule weight and activation movement. TinyTL brings the same concern to on-device learning, observing that fetching from DRAM costs roughly 100–200× the energy of an SRAM access. It argues for reducing training memory rather than parameter count. None of these treats optimizer state as a resident object with its own dynamics. Elsewhere in the file, the cost of that state appears only as a stated open problem: tripled storage in FP32.

On the optimization side, a clear body of work locates Adam's early instability in the empty second-moment estimate. At step one, an all-zero second moment turns Adam into pure sign descent. Abuduweili et al. and Revisiting the Initial Steps in Adaptive Gradient Descent Optimization fix this with data-driven or fan-in-scaled random initial second moments. RAdam attributes the problem to high variance in the adaptive step size while few samples have been seen, and switches adaptivity off until enough have accumulated. Kalra et al. tie warmup to sharpness. They propose initializing the second moment to the squared first gradient (GI-Adam). They also show that early gradient spikes stay locked into the second moment for thousands of steps when the averaging factor β₂ = 0.999. Two papers complicate the variance story. Ma et al. blame update magnitude rather than variance. Zhang et al. find the variance is finite and that heavy tails are the real issue. We take no side. Our design rebuilds per-weight magnitudes from history, which addresses the magnitude and heavy-tail views as well as the variance view.

The closest existing work is Memory-Efficient LLM Training with Dynamic Sparsity. It shows that regrowing pruned weights mid-training re-triggers the cold start. Because bias correction uses the global step count, regrown weights take steps about 3.16× larger than stable ones, and the loss spikes after every change to the network's structure. That paper motivates our hypothesis: cold starts recur whenever state is discarded. Our setting differs in what causes the discard and where the state goes. Here the discard is periodic and deliberate, a memory-capacity decision rather than a change to which weights exist. The design question is what to keep resident rather than how to initialize weights that never had state.

Core Claim

Our claim is that under a fixed SRAM budget, Adam can page its state block by block and nearly match unconstrained Adam. Two conditions are needed. First, each block keeps a persistent factored summary of its second moment: the row-wise and column-wise averages of the squared-gradient estimate, whose outer product rebuilds the full matrix of estimates. Second, each block's bias correction and RAdam rectification use the block's own step count rather than the global one. We further claim that this summary does better than the single-sample alternative. Rebuilding from the current squared gradient, GI-Adam style, inherits exactly the small-sample noise RAdam identifies. A summary of past history does not.

For a weight block of shape $r \times c$ with second-moment estimate $v$, we keep the row means $a_i = \frac{1}{c}\sum_j v_{ij}$ and column means $b_j = \frac{1}{r}\sum_i v_{ij}$. That is $r + c$ numbers instead of $rc$. On reactivation we set

$$\hat v_{ij} = \frac{a_i\, b_j}{\bar v},$$

where $\bar v$ is the block's overall mean of $v$. The first moment (the gradient average) restarts at zero. The block's local step counter $\tau$ resumes from its stored value rather than resetting, so bias correction reflects how much history the rebuilt $\hat v$ actually carries.

Research Program

We will build a paged Adam that runs over a partition of the model into blocks, normally one weight matrix or convolution per block. At any moment, $K$ blocks are active. Active blocks hold their full first and second moments in a simulated SRAM. Inactive blocks are frozen, and only their summaries and counters stay resident. Blocks rotate on a fixed schedule with period $P$ steps. Every byte crossing the simulated off-chip boundary is counted. The SRAM budget is expressed as a fraction of full Adam state: 50%, 25% and 10%.

On every reactivation we will log four quantities. The first is the spike in loss. The second is the ratio of each block's update size to that of its stable neighbors, the quantity Xiao et al. report as 3.16×. The third is the preconditioned sharpness measure of Kalra et al. The fourth is the relative error between the rebuilt $\hat v$ and the $v$ that unconstrained Adam would have held at that moment. Together these let us separate which failure explains any remaining gap: sign-descent collapse, wrong bias-correction scale, or wrong per-weight shape.

Baselines

The naive reference is paged Adam that resets both moments to zero on each reactivation and uses the global step count. That is what a straightforward implementation would do. We expect it to reproduce the post-regrowth spikes seen in dynamic sparse training. The quality ceiling is unconstrained Adam with tuned warmup, which ignores the memory budget. A second ceiling is exact paging: full state is written to DRAM and read back, giving identical math at the cost of maximal off-chip traffic. Our method should land near both ceilings in accuracy while moving a small fraction of exact paging's state traffic.

The strongest competitors are the published cold-start fixes, applied at every reactivation. These are RAdam rectification with a per-block counter, GI-Adam initialization from the squared current gradient, and the fan-in-scaled random second moment from Revisiting the Initial Steps. All three keep no persistent state, so all three are the most direct threat to our claim. We also include SGD with momentum, which needs half of Adam's state and so pages more cheaply.

Every method gets the same data, total step count, block partition, rotation schedule and SRAM budget. Learning rate and warmup length are each swept over the same grid with three seeds. We will claim success only under two conditions. Our method must beat the best per-reactivation fix on final metric at the 25% and 10% budgets, by more than seed-to-seed spread. It must also cut peak post-reactivation loss spikes.

Two ablations isolate the ingredients: summary with global counter, and local counter without summary. A third swaps the factored summary for a single scalar per block, to test whether per-weight shape matters or only overall scale does.

Why not just…?

Why not just use RAdam with a per-block step counter on every reactivation? This is the most serious alternative. It needs no new state, and RAdam's rectification is designed exactly for the few-sample regime. It does this by reverting to SGD-like behavior, however, so each reactivated block spends its early steps without per-weight scaling. With short rotation periods, a block may spend much of its active window that way. The "local counter without summary" ablation, crossed with RAdam, measures this directly. If it matches our method at all budgets, the summary is unnecessary and we will report that.

Why not just initialize the second moment from the squared current gradient, as GI-Adam does? It avoids sign descent at the cost of one gradient. But it rebuilds $v$ from a single sample, which is the high-variance estimate RAdam warns about. Kalra et al.'s long-memory finding means a heavy-tailed first gradient would suppress that block's steps for much of its window. The reconstruction-error and update-ratio logs will show whether this happens in practice. If GI-Adam matches us, the conclusion is that history does not matter at this rotation period.

Why not just drop Adam and use SGD with momentum? It sidesteps the problem, but the gaps in the file argue against it. The baseline is reported as tuned in RAdam and Revisiting the Initial Steps. On Transformers, adaptive methods are standard for a reason. We include it under identical budgets to measure that gap rather than assume it.

Why not just quantize the full optimizer state until it fits? This competes with our approach rather than replacing it. The file lists low-bit rounding of the second moment as an open brittleness, and no reported result was found on Lacuna for 8-bit optimizer state on these benchmarks. We treat it as a complementary axis. We run one condition that combines quantization with paging, but make no claims about quantization alone.

Proposed Experiments

We use the benchmarks where cold-start effects are documented. The first is IWSLT'14 De-En translation with a Transformer, where warmup sensitivity is largest. Revisiting the Initial Steps reports that vanilla Adam without warmup scores below 10 BLEU, and that its non-zero initialization reaches 31.53 BLEU without warmup and 33.71 with warmup, versus 30.14 for warmed-up Adam. The second is CIFAR-10 with a CNN, where that paper reports 96.02% versus 95.25% top-1. The third is ImageNet with ResNet, where RAdam reports 67.62% top-1 against Adam's 66.54%. These reported figures calibrate our unconstrained runs. Our own runs must first reproduce them within tolerance before any paged comparison is interpreted. For an on-device setting we add a transfer-learning task in the TinyTL style. The file gives no comparable training-accuracy number for it, so no reported result was found on Lacuna for that baseline.

The decisive check is a matrix of SRAM budget against rotation period $P$. If our hypothesis is right, the naive paged version should degrade sharply as $P$ shrinks, because cold starts become more frequent. The per-reactivation fixes should degrade more gently. Our method should stay flat until the budget leaves too few active blocks for the factored summary to track a $v$ that is still shifting. If the summary's advantage over GI-Adam and local-counter RAdam disappears at every $P$, the claim is falsified, and we will publish the negative result together with the traffic accounting. The traffic accounting is useful on its own, as the first measurement of optimizer-state traffic under an on-chip budget. A final run on an FPGA with on-chip-only memory, following the setup of Sung et al., would confirm that simulated traffic savings become energy savings. It is secondary to the simulated study.