Batch normalization rescales each layer's activations using the mean and variance of the current mini-batch. Two separate failures of this layer have been studied mostly in isolation. With tiny batches of one to four samples, the batch statistics are too noisy to use. When data arrives as a sequence of tasks, the running statistics drift toward the newest task, and old tasks are forgotten even if the weights barely change. Online continual learning on memory-limited hardware has both problems at once: a robot or edge device learns from a stream, one or two images at a time, with a small replay buffer. The usual fix for tiny batches is to pool statistics across recent training steps. Under a non-stationary stream, that pooling smooths over task boundaries and mixes incompatible distributions.
We propose a statistics memory for continual training at micro-batch sizes. It keeps a short history of per-layer means and variances and combines them as a weighted average. Each stored entry is corrected for how far the weights have moved since it was recorded, and it is down-weighted, rather than discarded, according to how far the data distribution appears to have moved. Because entries are tagged by source (incoming stream versus replay buffer), the same memory can also rebalance old-task and new-task statistics at no extra cost. Before building anything, the project asks whether this mechanism is needed at all: it first tests whether existing batch-independent continual normalization already holds up as the batch size shrinks to one.
Core Claim
A pooled normalization statistic built from past steps has three sources of error. Sampling variance falls as more history is used. Staleness from weight updates grows with the age of an entry and with the learning rate. Staleness from data drift grows with age whenever the task distribution shifts. Prior work handles at most two of these. Cross-Iteration Batch Normalization pools over a window of $k$ iterations and corrects weight staleness with a first-order Taylor term, but it assumes stationary data. Continual Normalization: Rethinking Batch Normalization for Online Continual Learning explicitly argues that such pooling compounds cross-task interference. MemBN pools test-time statistics in a queue and flushes it when a z-score test detects a shift, but it never corrects for weight updates and is evaluated only at test time.
Our claim is that the two staleness sources call for different treatments. Weight staleness is systematic and can be corrected. Data staleness is a real change in the target distribution and should reduce an entry's weight smoothly, not trigger an all-or-nothing flush. For layer $l$ at step $t$, the pooled mean is
$$\hat\mu_t=\frac{\sum_{\tau\in M} w_\tau\big(\mu_\tau+g_\tau(\theta_t-\theta_\tau)\big)}{\sum_{\tau\in M} w_\tau},\qquad w_\tau=n_\tau\, \pi_{s(\tau)}\exp(-D_\tau/T).$$
Here $M$ is the set of stored steps, $\mu_\tau$ is the mean recorded at step $\tau$, and $g_\tau$ is its gradient with respect to the layer weights, as in CBN; $\theta$ denotes those weights. The weight $n_\tau$ is the number of samples in the entry. The factor $\pi_{s(\tau)}$ is a balancing weight for the entry's source $s$ (stream or replay). $D_\tau$ is a standardized distance between the entry's corrected statistics and the current micro-batch's statistics, and $T$ is a temperature. The second moment is pooled the same way.
The layer sits in place of the batch-statistics stage inside CN's two-stage block. Group normalization still normalizes each sample on its own, and the memory restores the cross-sample statistics that a batch of one cannot supply. The source weights $\pi$ give a cheap analogue of the balancing in Rebalancing Batch Normalization for Exemplar-Based Class-Incremental Learning. TBBN balances by repeating and reshaping feature maps, which reportedly costs about 1.6x GPU memory and 1.3x time. We balance in the space of stored statistics instead. TBBN also balances gradients to the learnable scale and shift parameters, and our method does not. We treat that as a limitation to measure, not something we claim to solve.
Research Program
Phase 1: Is the problem real? We first measure how the methods closest to this problem degrade as the batch shrinks. These are standard BN, CN, and the recency-bias correction of Overcoming Recency Bias of Normalization Statistics in Continual Learning. We run them in online continual learning with experience replay at batch sizes 1, 2, 4 and 10. If CN's accuracy is flat across these sizes, its group-normalization stage is already doing the work and the project stops at a useful negative result.
Phase 2: Separate the error sources. We use controlled streams to pull apart sampling variance from loss of class diversity. Unraveling Batch Normalization for Realistic Test-Time Adaptation and Robust Test-Time Adaptation in Dynamic Scenarios blame small-batch failure on low class diversity. The stationary small-batch work, Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization and CBN, finds failure even with i.i.d. sampling. To test both views, we hold batch size fixed while varying class correlation within a task, and hold correlation fixed while varying batch size. In each condition we log three quantities per layer: the error of the pooled statistic against statistics computed on a large held-out batch under the current weights, the size of the Taylor correction, and the drift of the scale and shift parameters.
Phase 3: Build and ablate the layer. The ablations remove the Taylor correction, replace soft decay with MemBN-style hard flushing, and set $\pi$ to uniform. The temperature $T$ and window size are tuned once on Split CIFAR-10 and then frozen for all other benchmarks.
Baselines
The naive reference point is standard BN at each micro-batch size, run with experience replay and the same replay buffer and replay ratio as every other method. All methods share the same backbone, stream order, number of passes and buffer size. Each gets an equal hyperparameter search of twenty configurations on Split CIFAR-10.
The strongest existing competitors are CN and AdaB²N. CN reports roughly 3–4% higher average accuracy than BN on Split CIFAR-100, with about 3.7% overhead (Continual Normalization). AdaB²N reports up to 6.86% over BN on Split CIFAR-100 and up to 7.68% on Split CIFAR-10 (Overcoming Recency Bias). These results come from standard batch sizes. We found no reported result on Lacuna for any continual normalization method at batch sizes of four or fewer, so we will produce those numbers ourselves.
For the pooling mechanism in isolation, CBN is the reference. At batch size 2 on stationary ImageNet it reports 69.6% top-1, against 69.1% for GN and 55.9% for BN (Cross-Iteration Batch Normalization). We run it unchanged inside CN's block. For the exemplar-based class-incremental setting, the reference is TBBN, which reports final accuracy rising from 49.56% (BN) to 51.30% on ImageNet-100.
We count it as beating these baselines if the method exceeds the better of CN and AdaB²N by more than the across-seed standard deviation at batch sizes 1 and 2 on Split CIFAR-100 and Split Mini-ImageNet. It must also lose nothing at batch size 10. The key ablation is the same layer without the Taylor correction: if it matches the full method, then weight staleness is not the bottleneck in online continual learning, and we will report exactly that.
Why not just…?
Why not just use group normalization or CN, which do not depend on batch size? This is the most serious threat, and it is why Phase 1 comes first. The file has indirect evidence that batch-independent normalization leaves accuracy on the table. GN trails CBN at batch size 2 on ImageNet (69.1% vs 69.6%) and trails MemBN on ImageNet-C (69.38% vs 67.57% error). CN itself shows that adding cross-sample statistics on top of GN helps continual learning. Whether that benefit survives at batch size 1 in continual learning is unknown, and the Phase 1 sweep answers it directly.
Why not just pool uncompensated history, as Momentum BN or MemBN do? Online continual learning takes few, small gradient steps, so weight staleness may be negligible. Both Momentum Batch Normalization and MemBN work without any correction. CBN reports degradation without compensation, but that was under stationary, large-step training. The no-Taylor ablation decides this question, and if it wins we drop the correction, which removes CBN's roughly 11% extra FLOPs.
Why not just enlarge the replay batch or use synchronized BN? In our target setting the limit is activation memory on a single device. A larger replay batch therefore violates the constraint, and synchronized BN needs devices that do not exist. To be fair to this option, we report it as an upper-bound reference that ignores the memory limit, not as a competitor.
Why not just switch to switchable normalization? It learns a mixture of instance, layer and batch scopes, and it held 75.6% top-1 at batch size 2 on ImageNet (Switchable Normalization). However, it adds about 29% training overhead in continual learning, and its batch scope still uses the current micro-batch. We include it once at batch size 1 on Split CIFAR-100 as a check.
Proposed Experiments
The main benchmarks are online Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, reporting average accuracy, forgetting and wall-clock overhead at batch sizes 1, 2, 4 and 10 over five seeds. A long-tailed check on COCOseq tests whether soft decay copes better than flushing with imbalanced streams. ImageNet-100 class-incremental learning tests the source-balancing weights against TBBN, comparing both accuracy and GPU memory.
To test the moving-target question in its original habitat, we also run gradient-based continual test-time adaptation on CIFAR-10-C at batch size 1. There, MemBN reports 12.88% error and TEMA reports 21.05% error. If the compensated memory beats MemBN there, the weight-staleness argument holds beyond continual learning.
Three outcomes would be decisive. The first is whether CN is flat across batch sizes. The second is whether the Taylor correction matters once the data drifts. The third is whether, in the Phase 2 streams, the error in pooled statistics follows batch size or class diversity.
References
- Cross-Iteration Batch Normalization
- Continual Normalization: Rethinking Batch Normalization for Online Continual Learning
- MemBN: Robust Test-Time Adaptation via Batch Norm with Statistics Memory
- Rebalancing Batch Normalization for Exemplar-Based Class-Incremental Learning
- Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation
- Unraveling Batch Normalization for Realistic Test-Time Adaptation
- Robust Test-Time Adaptation in Dynamic Scenarios
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
- Momentum Batch Normalization for Deep Learning with Small Batch Size
- Differentiable Learning-to-Normalize via Switchable Normalization