Stateful Layer Normalization for Streaming Test-Time Adaptation

Vision Transformers and other attention models use layer normalization, which rescales each token using only that token's own numbers. That makes them immune to small batches. It also means they keep no record of what their training data looked like. Convolutional networks use batch normalization, which keeps running averages over many inputs, and those averages became the hook for a large family of cheap adaptation tricks. When a deployed model meets fog, sensor noise or a new scanner, you can re-estimate the stored averages from incoming unlabeled data and recover much of the lost accuracy without any gradient steps. Transformers have no such hook. The one backprop-free method that works with layer normalization, LN-TTA, recalibrates only the final scale and shift of each layer, and its authors report that it remains sensitive to test batch size. It was also designed for a single fixed corruption rather than a stream that keeps changing.

We propose adding a small memory to layer normalization at test time. The memory leaves layer normalization's per-token standardization untouched. Next to it, the layer keeps a compact running record of per-channel statistics of its standardized outputs, and uses that record to set its scale and shift without backpropagation. The record is built from scalars, never from stored images or features. How far back it looks is set by a bias–variance rule rather than a fixed decay or a hard reset: the rule shortens the memory when the stream drifts fast and lengthens it when single samples are too noisy to trust. The goal is a Transformer that adapts with batch size 1 across long chains of domain shifts.

Core Claim

Two results in the literature conflict, and both can be accommodated. Revisiting Layer Normalization for Point Cloud Test Time Adaptation argues that layer normalization's mean removal and radius normalization already absorb much of a domain shift. On that view, re-standardizing across a batch is unnecessary and can hurt, and adaptation belongs in the post-affine gain and bias. The memory-based batch-norm methods, MemBN and Robust Test-Time Adaptation in Dynamic Scenarios, argue that reliable adaptation needs statistics pooled over far more samples than one test batch contains.

Our claim is that the two are compatible because they act at different points in the layer. Keep the pre-affine step sample-local, as LN-TTA recommends. Then estimate the post-affine correction from a temporally pooled memory, as MemBN does. This should remove LN-TTA's batch-size sensitivity. It should also beat fixed-decay pooling on non-stationary streams, because the memory depth tracks the measured drift.

The testable prediction has two parts. First, at batch size 1 on continual corruption streams, memory-backed post-affine calibration should approach LN-TTA's large-batch accuracy. Second, an adaptive window should beat the best fixed exponential moving average (EMA, a running average with a fixed forgetting rate) chosen in hindsight for each stream, not only an untuned one.

Mechanism

For each layer-norm site $\ell$, let $z$ be the standardized token vector before the affine step. The memory holds a short ring of per-step summaries: the per-channel mean and second moment of $z$ over that step's tokens, plus a token count. Storage is a few kilobytes per layer. Offline, we record the same per-channel summaries once on source data. This gives the Transformer the "statistical diary" that batch normalization keeps implicitly, which Gong et al. and Li et al. identify as the missing ingredient for porting batch-norm techniques to Transformers. At test time, the layer sets its gain and bias by closed-form first- and second-moment matching. Pooled target moments from the memory are mapped onto the source moments, with the same moment-matching form LN-TTA uses.

The new piece is how many past steps to pool. Suppose a channel's true target mean drifts by $\delta$ per step. Suppose also that one step contributes $n$ tokens with per-token spread $s$. Averaging the last $w$ steps then has squared error of roughly

$$ \mathrm{MSE}(w) \approx \frac{s^2}{n\,w} + \left(\frac{\delta\,w}{2}\right)^2, \qquad w^\* = \left(\frac{2 s^2}{n\,\delta^2}\right)^{1/3}. $$

The first term is sampling variance shrinking with pooled sample count. The second is the lag bias of a window centered $w/2$ steps in the past under linear drift. We estimate $s^2$ from within-step spread and $\delta$ from the gap between a short-window and a long-window mean, both per layer. The window is then set by $w^\*$, clipped to the ring length. An abrupt shift makes $\delta$ large, which shrinks the window to a few steps. This replaces MemBN's all-or-nothing Z-score flush with a graded response. It is also more explicit than the shift-aware forget gate in MECTA, which modulates long- versus short-term statistics but is defined on channel-wise batch statistics in CNNs.

An optional second ingredient handles streams where one class arrives in long runs. Here the memory is split into slots keyed by the model's predicted class, and pooled moments weight the slots equally. This borrows the idea behind UnMix-TNS and RoTTA's class-balanced buffer, applied to scalars rather than samples or batch-norm components. It is ablated separately and is not part of the core claim.

Research Program

The work has three parts.

The first is building the layer and checking a staleness question. Adapting an upstream gain changes the inputs to downstream layers. Memory entries recorded earlier were produced under different upstream parameters, which is the moving-target problem that Cross-Iteration Batch Normalization handles for batch norm with a Taylor correction. We expect layer normalization's per-token standardization to remove much of this mismatch, because it cancels common shifts and scales. We will measure the effect directly. For a frozen set of inputs, we compare memory statistics recorded under the parameters of earlier steps against statistics recomputed under current parameters, per layer and by depth. If the gap is material in deep layers, consistent with Continual Normalization's finding that variance-estimation error grows with depth, we will add a cheap correction. That correction re-scales stored moments by the ratio of current to recorded upstream gains.

The second part is the head-to-head on LN-TTA's own benchmarks: ScanObjectNN-C, ModelNet-C and ShapeNet-C. We sweep batch size from 1 to 64 and run both stationary and chained-corruption orderings.

The third part extends to image Transformers on continual corruption streams and to a temporally correlated video benchmark. This tests whether the bias–variance window generalizes beyond point clouds.

Baselines

The naive reference is the frozen source model with no adaptation, together with plain per-batch LN-TTA, which estimates moments from the current batch only. Plain LN-TTA reports mean accuracy +12.35% over Source Only on ScanObjectNN-C and +15.58% on ModelNet-C in Revisiting Layer Normalization for Point Cloud Test Time Adaptation. We will reproduce it with the authors' setting and then shrink the batch.

The strongest competitor on the actual question is a fixed-decay EMA of the same per-channel moments feeding the same closed-form calibration. Its decay is tuned by grid search on held-out streams. We also report an oracle variant with decay chosen per test stream in hindsight. This is the natural "simple fix" for LN-TTA's batch sensitivity. It uses identical memory, identical calibration and identical compute, so it isolates the adaptive window. Beating it means lower average error than the held-out-tuned EMA on chained streams. Matching the oracle-tuned EMA without per-stream tuning also counts.

A second strong competitor is a MemBN-style FIFO queue with Z-score flush, ported to our post-affine moments. It is given the same queue length and a tuned threshold. It tests graded versus hard forgetting.

For gradient-based reference points, we run CFA. It aligns ViT features to offline source moments up to fourth order and reports 19.8% top-1 error on ImageNet-C with BeiT-L16, but in single-domain evaluation. We also run TENT in continual mode. Both get the same backbone, stream order and a fixed tuning budget per method. We report peak GPU memory and throughput alongside error, since LN-TTA's 1.468 GB versus TENT's 2.890 GB on ScanObjectNN-C is part of the case for backprop-free methods.

For CNN context only, we cite Continual Test-Time Domain Adaptation, whose CoTTA reaches 16.2% average error on CIFAR-10-C, and MemBN's 12.88% at batch size 1 on CIFAR-10-C. Architectures differ, so these are not claimed as direct comparisons.

Ablations remove each piece of the proposal in turn. One adapts the pre-affine statistics across the memory, which tests LN-TTA's harmfulness claim in the streaming setting. Others remove the class-keyed slots and the staleness correction.

Why not just…?

Why not just use a fixed EMA on LN-TTA's moments? This is the real threat. A well-chosen decay might capture most of the benefit. The bias–variance analysis predicts that any single decay is wrong on streams that mix slow drift with abrupt jumps. It does not guarantee that the error is large. The decisive comparison is against the oracle per-stream decay on chained corruptions with varied segment lengths. If the adaptive window does not beat the held-out-tuned EMA and come close to the oracle, we will report that the contribution is the memory itself plus the staleness analysis, not the window rule.

Why not just port MemBN's queue and reset unchanged? MemBN works on batch statistics in CNNs and flushes its whole queue on a Z-score trigger. The file notes this risks dropping useful support too early under gradual drift, or keeping corrupted statistics too long. Our queue-with-flush baseline uses the same post-affine target. Any gap therefore isolates graded versus binary forgetting, and is not an artifact of where the statistics live.

Why not just re-standardize layer normalization across the memory, like batch norm? LN-TTA argues this is harmful, and it is the most direct reading of Gap 1. We include it as an ablation rather than assuming the answer. If cross-memory pre-affine standardization helps under continual streams, that would contradict LN-TTA's single-domain conclusion, which would itself be a useful finding.

Why not just run CFA or TENT continually? Both backpropagate, which raises memory cost. CFA was evaluated on isolated corruptions. Continual Test-Time Domain Adaptation shows continual TENT degrading to 20.7% error on CIFAR-10-C through error accumulation. We run both on the same streams and report error against memory. A backprop-free method that only ties them on error at a fraction of the memory is still a useful result.

Proposed Experiments

The primary experiment uses ScanObjectNN-C and ModelNet-C with LN-TTA's point-cloud Transformer. It runs batch sizes 1, 4, 16 and 64 under stationary per-corruption evaluation and under a chained ordering that cycles through all corruptions several times. Metrics are mean accuracy, accuracy at the end of the final cycle versus the first (forgetting), throughput and peak memory.

The decisive check has two parts. At batch size 1, the memory-backed method should recover most of the gap between per-batch LN-TTA at batch 1 and at batch 64. On chained streams, it should beat the held-out-tuned EMA.

The secondary experiment is a ViT on CIFAR-10-C and ImageNet-C under the CoTTA-style continual protocol. Error is reported against CFA and continual TENT. We found no reported continual-stream ViT result on Lacuna for ImageNet-C, so all numbers there are new.

The label-correlation experiment uses ImageNet-VID-C with a ViT backbone. It tests whether class-keyed scalar slots recover the kind of gain UnMix-TNS reports for batch norm (23% accuracy over test-time BN). We found no reported result on Lacuna for a layer-norm model on this benchmark.

A final diagnostic plots the chosen window $w^\*$ per layer over time against the known shift boundaries of the stream. It also plots the measured staleness gap by depth. These show whether the mechanism behaves as the analysis predicts, beyond whether it improves accuracy.