Unbiased Gradients, Biased Curvature: Correcting Preconditioners Trained on Compressed Activations

Two memory-saving ideas in neural network training are hard to combine. The first is activation compression: during the forward pass, the intermediate outputs of each layer ("activations") are stored in 2–4 bits instead of 16 or 32, which lets much larger models or batches fit in memory. The second is second-order preconditioning, as in optimizers like Shampoo, which rescale each gradient step by an estimate of the loss curvature so that steep and flat directions move at similar speeds. Activation compression rests on a guarantee: if values are rounded randomly up or down in proportion to their distance from each grid point, the resulting gradient is correct on average. Curvature statistics do not inherit that guarantee. They are built from products of gradients with themselves, and the average of a squared noisy quantity equals the square of the true quantity plus its variance.

This note proposes to measure that bias, derive it in closed form, and remove it cheaply during training. For a linear layer trained on randomly rounded activations, the extra term in Shampoo's two curvature statistics has an exact expression. It depends only on quantities already present in the backward pass: the known rounding variance of each stored activation, and the gradient flowing back into the layer. The bias is not a harmless uniform damping term. It is concentrated on the input features with the widest ranges, which are exactly the outlier channels that low-bit quantizers struggle with. It also inflates the small eigenvalues that an inverse matrix root amplifies most. We expect it to erode much of Shampoo's advantage over Adam precisely when activations are compressed aggressively. A subtraction costing roughly as much as forming the statistics themselves should restore that advantage.

Core Claim

Consider a linear layer with stored input activations $A \in \mathbb{R}^{b \times m}$ (batch $b$, $m$ input features) and backpropagated signal $\Delta \in \mathbb{R}^{b \times n}$. The exact weight gradient is $G = A^\top \Delta$. Under stochastic-rounding compression, the stored activations become $\tilde A = A + E$, where $E$ has independent zero-mean entries with variance $\sigma^2_{ik}$. For a value $x$ between grid points $\ell$ and $u$, that variance is $(x-\ell)(u-x)$, so it is known at compression time. The compressed gradient $\tilde G = G + E^\top \Delta$ is unbiased, as ActNN shows. Shampoo, however, accumulates the left and right statistics $L = GG^\top$ and $R = G^\top G$, and their expectations are shifted:

$$\mathbb{E}[\tilde G \tilde G^\top] = GG^\top + \mathrm{diag}_k\Big(\textstyle\sum_i \sigma^2_{ik}\,\|\delta_i\|^2\Big), \qquad \mathbb{E}[\tilde G^\top \tilde G] = G^\top G + \textstyle\sum_i s_i\, \delta_i \delta_i^\top,$$

Here $\delta_i$ is the $i$-th row of $\Delta$, and $s_i = \sum_k \sigma^2_{ik}$ is the total rounding variance of sample $i$. Both correction terms follow from independence of the rounding noise: only matched indices survive the expectation.

The left correction is diagonal and costs $O(bm)$ to compute. The right correction is a weighted Gram matrix of $\Delta$ and costs the same as forming $R$ itself. The same argument gives an elementwise correction for Adam's second moment.

We claim three things. First, at 2–4 bits this bias measurably distorts Shampoo's preconditioner, and most strongly on outlier-heavy input channels. Second, the distortion makes second-order optimizers lose more quality under activation compression than first-order optimizers do. Third, subtracting the bias inside the running average of the statistics recovers most of that loss. It should do so more effectively than any retuning of the scalar damping constant, because the bias varies across features and across layers.

Relation to Prior Work

The motivating evidence comes from two sides.

ActNN compresses stored activations to an average of 2 bits, with per-layer bit-widths chosen to minimize gradient variance. It proves the gradient is unbiased and reports 76.9% top-1 accuracy on ImageNet ResNet-50 against 77.1% at full precision. It evaluates only first-order optimizers and never examines second moments of the compressed gradient. Backprop with Approximate Activations likewise keeps the forward pass exact and compresses only what is saved for the backward pass. Its setting is also first-order.

HAWQ-V2 and the Kronecker-Fisher sensitivity work compute curvature offline, with full-precision activations, in order to decide how to quantize a finished model. We instead study curvature estimated during training from activations that are already quantized.

The Kronecker-Fisher paper also constrains our design. It reports that local quadratic curvature underestimates sensitivity on output projections dominated by large residual-stream outliers. Our formula predicts that those same channels, which have the widest quantization ranges, receive the largest spurious diagonal mass. Correcting only the average would therefore miss the point. Value-aware Quantization supplies a natural companion technique: keep a small fraction of outliers in full precision so that their variance is zero. We use it as an ablation, not as the core mechanism.

The optimizer we modify, and the step-time accounting we follow, come from DASH. DASH batches Shampoo's inverse-root computations and runs its Coupled-Newton solver in FP16, cutting step time from 666ms to 138ms at a 1K batch size. It leaves activation memory untouched. It also observes that some of its more accurate solvers become unstable in FP16 on ill-conditioned matrices.

That observation creates a real tension for our proposal. The compression bias acts like accidental damping that improves conditioning. Removing it may therefore make a low-precision inverse-root solver less stable. We treat damping as a separate, deliberately chosen quantity, set after debiasing, and we report solver stability explicitly.

RSO points in a different direction: it projects activations into a random subspace and suggests running second-order methods inside that subspace. That obvious intersection is not what we do. We keep full-rank Shampoo and fix the statistics it is fed.

Research Program

The first stage is diagnostic. On a fixed checkpoint, in the same step, we compute Shampoo's statistics twice: once from exact activations and once from activations compressed at 2, 3, 4 and 8 bits using ActNN's per-group ranges. We then compare three things. The first is the measured bias against the closed-form prediction. The second is the eigenvalue spectra of both statistics. The third is the resulting inverse fourth roots, measured as the angle between preconditioned update directions.

This stage also tests the independence assumption, since ActNN's per-group scaling couples entries within a group. It tells us which layers and channels carry the bias. We expect the bias to be negligible at 8 bits and dominant at 2 bits. If it is small even at 2 bits, the project stops here and reports a negative result.

The second stage builds the correction. At backward time, the compressed-activation kernel already knows each element's grid interval. We emit the per-sample total variance $s_i$ and the per-feature weighted sums $\sum_i \sigma^2_{ik}\|\delta_i\|^2$. We subtract both from the exponential moving averages of $L$ and $R$.

Subtracting an expectation from finite-sample running averages can produce small negative eigenvalues. We will clip at the damping floor before the inverse root, and we will report how often clipping activates. Clipping should be rare, because subtraction happens inside the average rather than per step.

The same machinery gives a debiased Adam second moment, which serves as the first-order control. Implementation is in the DASH batched Shampoo codebase, so step-time comparisons match DASH's accounting.

The third stage, run only if the first two succeed, revisits bit allocation. ActNN chooses per-layer bits to minimize raw gradient variance. A preconditioned step, however, amplifies variance along low-curvature directions. A variance objective weighted by the preconditioner may therefore assign bits differently, and we will measure whether it does.

Baselines

The naive reference point is standard mixed precision in the style of Mixed Precision Training: BF16 execution with FP32 master weights, uncompressed activations, and AdamW. It anchors both memory use and quality.

The second reference is AdamW with ActNN compression. For ImageNet ResNet-50, the file gives ActNN's reported 76.9% top-1 at 2 bits against 77.1% at full precision. We rerun this under our training budget and do not borrow the number.

The strongest existing method is DASH's Shampoo with exact activations. It is the quality ceiling the proposal is trying to approach at compressed-activation memory. The file reports DASH's optimizer-step speedup and notes that its NDB solver reaches the lowest validation perplexity on C4. It gives no absolute perplexity. On the GPT-2 124M FineWeb setting, the closest reported matrix-preconditioner result is PRISM's validation loss of 4.98, against AdamW's 6.86 at equal training time. We will reproduce our own full-activation Shampoo number rather than rely on that figure.

The direct competitor is naive Shampoo on ActNN-compressed activations with no correction. For Shampoo combined with activation compression, no reported result was found on Lacuna.

The two ablations that isolate the key ingredient are as follows. The first is naive Shampoo with a retuned global damping constant. The second is the stronger version: per-layer damping tuned by sweep. Because debiasing removes a feature-dependent diagonal term, this baseline tells us whether that feature dependence actually matters.

Further ablations debias only $L$ or only $R$, and add outlier preservation in the style of Value-aware Quantization on top of the naive method.

All methods share data, token or epoch budgets, model, and bit allocation. Each receives the same number of hyperparameter trials over learning rate and damping. The proposal counts as beating the competition under two conditions. First, debiased Shampoo must close at least half of the quality gap between naive compressed Shampoo and exact-activation Shampoo. Second, it must beat per-layer tuned damping by more than seed-to-seed variation across three seeds, while its step time stays within a few percent of naive compressed Shampoo.

Why not just…?

Why not just increase the damping constant? The bias looks like damping, so a tuned constant might absorb it. That is the most serious threat to the proposal. Our formula says the bias is proportional to per-feature quantization range, which differs by orders of magnitude between outlier and ordinary channels. A scalar cannot match it, and even a per-layer scalar cannot match variation within a layer. The per-layer-damping baseline settles this. If it matches debiasing, we will report that the bias is benign in practice and redirect the work toward diagnostics and bit allocation.

Why not just use gradient checkpointing, or compress activations only to 8 bits? Both avoid the bias: checkpointing recomputes exact activations, and at 8 bits the rounding variance is tiny. According to ActNN, checkpointing costs roughly 30–40% extra compute per step. ActNN's order-of-magnitude memory savings exist only at 2–4 bits, which is exactly where the bias is largest. We include checkpointed exact-activation Shampoo as a matched-memory baseline and report quality against wall-clock time. Debiasing is justified only if it beats checkpointing on that axis.

Why not refresh the curvature statistics from exact activations every few steps? Shampoo already amortizes its inverse roots. One could accumulate statistics only on occasional steps that recompute exact activations. This is a credible alternative that trades staleness for exactness. We run it with the refresh interval chosen so that its compute overhead equals our correction's overhead. The comparison shows whether fresh-but-biased statistics, once corrected, beat stale-but-exact ones.

Why not just use Adam, which ActNN already shows works? Adam's second moment is biased too, just more mildly, and we debias it as a control. The claim at stake is that second-order methods lose disproportionately under compression. If naive Shampoo's compression penalty is no larger than AdamW's, the motivating claim fails, and we will say so.

Proposed Experiments

We run three settings of increasing cost. The first is ResNet-20 on CIFAR-10, for the full sweep over bits and ablations. The second is ResNet-50 on ImageNet, at a reduced epoch budget, for comparison with ActNN. The third is GPT-2 124M on FineWeb, following PRISM's setup, with LLaMA-style small models on C4, following DASH, as a replication.

Metrics are top-1 accuracy or validation loss at fixed tokens and at fixed wall-clock time, peak memory, optimizer step time, and FP16 solver failure rate.

The first decisive check is the stage-one diagnostic. The measured bias in $L$ and $R$ must match the closed form, and it must be dominated by high-range channels. The second check is the compression penalty: the drop in quality from exact to 2-bit activations must be clearly larger for Shampoo than for AdamW. The third check is the headline comparison against per-layer tuned damping, checkpointing, and periodic exact refresh at equal compute.

A final stability check runs debiased Shampoo with DASH's FP16 Coupled-Newton solver. It asks whether removing the accidental damping reintroduces the low-precision breakdowns DASH reports, and whether explicit damping restores stability.