Cheap Statistics Memory for Continual Normalization at Batch Size One

Many networks normalize each layer's activations using the mean and variance of the current mini-batch. This is batch normalization, and it works well when batches are large and drawn from a fixed data distribution. Online continual learning breaks both conditions. The model sees a stream of tasks one after another, often under tight memory limits that force updates on only one or two images at a time. The best normalizer currently designed for continual learning, Continual Normalization, still contains a batch-normalization stage. At batch size one or two, that stage computes its statistics from almost no data. We propose to replace those statistics with a short memory of recent per-step statistics. Each entry is tagged by whether it came from the incoming stream or from the replay buffer (a small store of past examples that is mixed into training), and the entries are pooled in task-balanced proportions rather than by recency.

The claim that makes this worth testing is mechanistic, not just additive. Continual Normalization first applies group normalization, which rescales each sample on its own. We hypothesize that this first stage absorbs most of the drift that makes historical statistics stale. If so, the second-stage statistics change slowly enough across weight updates and even across task boundaries that a plain queue works. The expensive gradient-based correction used by Cross-Iteration Batch Normalization would then be unnecessary. If the hypothesis holds, memory-augmented normalization becomes cheap enough for continual learning on small devices. If it fails, the staleness measurements will show where and why it fails.

Core Claim

Inside Continual Normalization, statistics cached over a short window of recent steps stay close to the statistics the current weights would produce, much closer than the same cache inside plain batch normalization. Pooling those cached statistics with explicit stream/replay balance recovers the accuracy of large-batch Continual Normalization at physical batch sizes of one to four. It does so without Taylor compensation and without per-task momentum schedules.

Where the Pieces Come From

Two papers define the continual-learning side. Continual Normalization shows that batch normalization's running statistics drive forgetting. It replaces batch normalization with group normalization followed by a batch normalization without learned scale. It reports roughly 3–4% average-accuracy gains on Split CIFAR-100, at a 3.7% runtime cost. Its experiments use batches of ten or more, so its second stage is never stressed at micro-batch sizes. That untested stage is the gap this proposal exploits. AdaB²N attacks the same recency bias with a different tool. It models the task mixture within each batch with a Dirichlet distribution and uses an inference recurrence that interpolates between an exponential moving average and a cumulative average. It reports gains over batch normalization of up to 7.68% on online Split CIFAR-10 and 6.86% on Split CIFAR-100. It also assumes within-batch statistics are reliable. We borrow its insight that balance, not recency, should govern pooling. We apply it at the level of tagged memory entries instead of Dirichlet weights inside a single batch.

The memory side supplies the method and the main risk. Cross-Iteration Batch Normalization pools statistics over the last $k$ steps. It corrects them toward the current weights with a first-order Taylor term, which costs about 11% extra computation and 7% extra memory. On ImageNet at batch size 2 it reaches 69.6% top-1, against 55.9% for batch normalization and 69.1% for group normalization. Without that correction, its authors find naive averaging fails. That finding is the central threat to our hypothesis, and we test it directly rather than assume it away. Momentum Batch Normalization and MABN use moving averages of statistics during training, in the forward and backward passes respectively. Both were evaluated only on stationary data, where recency is harmless. MemBN, TEMA and RoTTA use statistics queues, shift-triggered resets and class-balanced memory banks in non-stationary streams. All three work in unsupervised test-time adaptation, where weights barely move. Supervised continual training takes large gradient steps, which is exactly the regime in which cached statistics go stale.

Research Program

The layer, which we call memory-backed Continual Normalization, keeps group normalization as the first stage. In the second stage, each step pushes the per-channel mean and second moment of the current physical batch into a queue of length $k$. Each entry is tagged with its source (stream or replay) and with its source task, which is known for replay samples and inferred from the current task for stream samples. The normalizing mean used at step $t$ is

$$\hat\mu_t = \sum_{s} w_s \, \frac{1}{|Q_s|}\sum_{j \in Q_s} \mu_j ,$$

where $Q_s$ is the set of queued entries belonging to group $s$ (the current stream task, or one past task represented in replay), $\mu_j$ is an entry's mean, and $w_s$ are balance weights. The weights default to uniform over seen tasks. A recency-weighted variant is included as an ablation. The variance is pooled the same way from second moments. In the backward pass, only the current batch's contribution carries gradient and past entries are constants, the same treatment Cross-Iteration Batch Normalization uses. Inference uses running statistics accumulated with the same balanced pooling.

The analysis component measures staleness directly. At checkpoints, we recompute each queued entry's statistics with the current weights on the same stored inputs and record the normalized gap to the cached value. We measure this gap separately within tasks and across task boundaries, comparing the second stage of Continual Normalization against an ordinary batch normalization layer at the same depth. This tests the core claim without relying on accuracy alone. An optional Taylor-corrected variant is built so we can measure what the correction buys inside each normalizer.

Baselines

The naive reference is standard batch normalization at the micro-batch size, together with unmodified Continual Normalization at the same size. Comparing the two quantifies how much of Continual Normalization's published advantage survives at batch size one to four. No reported result was found on Lacuna for Continual Normalization or AdaB²N below batch size ten, so we will produce these numbers ourselves.

Pure group normalization is the batch-independent baseline. It is immune to batch size by construction, so it is the most natural thing a reviewer would expect to win in this regime.

The strongest existing small-batch method is Cross-Iteration Batch Normalization. We drop it into the second stage of Continual Normalization and, separately, use it in place of batch normalization. Its best reported figure is 69.6% top-1 on ImageNet at batch size 2, against 69.1% for group normalization. No reported result was found on Lacuna for it in continual learning.

The strongest continual-learning normalizer is AdaB²N. Its best reported result is up to +6.86% average accuracy over batch normalization on online Split CIFAR-100. We run it at the same micro-batch sizes.

Momentum Batch Normalization in the second stage tests whether a single moving average suffices. Gradient accumulation and replay padding are run as fairness controls, described below.

Every method uses the same backbone (a reduced ResNet-18, as is standard on these benchmarks), the same replay method (experience replay and DER++), the same buffer sizes and the same peak activation memory. Hyperparameter search uses an equal number of trials on the first two tasks only, covering window $k$, momentum, $\kappa$ and group count. Beating a baseline means higher final average accuracy and lower forgetting across three or more seeds, with non-overlapping confidence intervals, at no more than 5% extra runtime over Continual Normalization.

Three ablations isolate the key ingredients. Removing the tags (one recency-weighted queue) tests whether balance matters. Adding the Taylor correction tests whether the cheap version loses anything. Moving the queue onto plain batch normalization tests whether the group-normalization stage is what makes the queue safe.

Why not just…?

Why not just use group normalization and drop batch statistics entirely? It is batch-independent and costs nothing extra. Continual Normalization reports that it underperforms Continual Normalization with replay, attributing the gap to lost cross-sample coupling. Proxy Normalization documents channel-wise collapse in deep group/layer-normalized networks. At micro-batch sizes, however, that cross-sample coupling may be worth little, so group normalization is a genuine threat. It is the first comparison we report. If it matches our method at batch size 1, the right conclusion is that the batch stage should simply be removed in this regime.

Why not just pad each step with more replay samples? When the constraint is latency or stream rate rather than memory, padding a single stream image with 31 replay images gives a real batch for free. That is a real competitor. We therefore separate two regimes. In the memory-bound regime, total physical batch size is capped, so padding is infeasible and our method is needed. In the stream-bound regime, padding is allowed and is reported as a baseline. We expect padding to win or tie there, and we will say so.

Why not just accumulate gradients? Accumulation averages gradients but leaves every forward pass normalized by one or two samples. MABN and Momentum Batch Normalization identify exactly this forward noise as the failure mode. Accumulation also delays updates in an online stream. We include it at matched memory to confirm this empirically.

Why not just use Cross-Iteration Batch Normalization, or AdaB²N, as published? The first pools across task boundaries by recency and pays for a Taylor correction. The second balances tasks but computes within-batch statistics that are meaningless at batch size 1. Our ablations, which add the Taylor term back and remove the tags, show which of these two missing pieces accounts for any gain.

Decisive Checks

We run online class-incremental Split CIFAR-10, Split CIFAR-100, Split Mini-ImageNet and Split Tiny-ImageNet at physical batch sizes 1, 2 and 4. For each run we report final average accuracy, the forgetting measure, wall-clock time and peak memory. The hypothesis holds if three things are true. First, the measured staleness gap in the second stage of Continual Normalization is several times smaller than in plain batch normalization, including across task boundaries. Second, adding Taylor correction to our layer changes accuracy by less than seed noise. Third, our layer at batch size 2 comes within one point of unmodified Continual Normalization at batch size 10 while beating group normalization.

The hypothesis is falsified if any of three things happens. Staleness inside Continual Normalization may be as large as in batch normalization. Taylor correction may be required to beat group normalization. Or untagged recency pooling may match the tagged version, which would mean the continual-learning contribution is empty.