Gain-Modulated Overcomplete Normalization for Streaming Video Adaptation

Video models built on transformers and state-space models use Layer Normalization, which rescales each token using only that token's own activations. This makes them robust to batch size, but it also means they store no record of what typical features looked like during training. Convolutional networks keep such a record in their Batch Normalization statistics, and a whole family of cheap test-time fixes depends on it: re-estimating those statistics on incoming data, or shrinking them toward the training values. None of these fixes carries over to modern video backbones. We propose replacing Layer Normalization with a layer borrowed from a model of sensory adaptation in the brain. A fixed, overcomplete dictionary (more feature directions than channels) is learned slowly during training. A small vector of per-direction gains adapts quickly at test time, and these gains whiten the features, removing their correlations, from moment to moment.

The gains do three jobs at once. Their training-time values act as a stored population record, the way Batch Normalization statistics do. Because there are more directions than channels, adjusting them can track changes in feature correlations, not only in per-channel variance. And because they can be updated at several speeds, they provide a natural hierarchy over time for streams whose shifts arrive at the scale of seconds, scenes, or hours. The two-sentence version is this: give LayerNorm-based video models a slow overcomplete dictionary plus fast gains, so they have something principled to adapt at test time. Then run those gains at several timescales so the model survives streams where the same content persists for a long stretch.

Core Claim

The mechanism comes from Adaptive whitening with fast gain modulation and slow synaptic plasticity. That paper decomposes an inverse whitening matrix as $M = \alpha I + W\,\mathrm{diag}(g)\,W^\top$. Here $W \in \mathbb{R}^{d \times K}$ holds $K > d$ dictionary directions over $d$ channels, $g \in \mathbb{R}^K$ holds per-direction gains, and $\alpha$ is a small ridge constant. A token feature $x$ maps to the whitened response $r = M^{-1}x$. The gains are driven toward unit variance along each direction:

$$\Delta g_i \propto \mathbb{E}\big[(w_i^\top r)^2\big] - 1,$$

where $w_i$ is the $i$-th column of $W$ and the expectation is over recent tokens. That paper only tested this on 2D Gaussians and 5×5 image patches. Our claim is that it scales into a normalization layer for deep video networks. We expect three properties. First, it matches LayerNorm accuracy on clean data. Second, it supports Batch-Normalization-style test-time adaptation on LayerNorm architectures, where NOTE and others note such adaptation is unavailable. Third, when its gains are split across timescales and anchored to their training values, it avoids the "over-whitening" collapse that NOTE documents for temporally correlated streams.

Overcompleteness is what makes this more than a renamed per-channel variance tracker. With $K = d$ and $W = I$, the gains can only rescale channels. With $K > d$, a $K$-dimensional gain update reshapes the effective covariance in many directions while still costing only $K$ scalars per layer at test time. That is far cheaper than estimating and inverting a $d \times d$ covariance online.

Research Program

We build the layer, which we call GainNorm, as a drop-in replacement for LayerNorm. We fit it into two architectures: UniFormer, which uses local 3D convolutions in shallow layers and global attention in deep ones, and VideoMamba, a linear-time bidirectional state-space model. We use two backbones so the result is about normalization rather than one architecture. The forward pass computes $M^{-1}x$ with a few conjugate-gradient steps, or with a learned feed-forward approximation that we distill and check against the exact solve.

During training, $W$ and a source gain vector $g_{\text{src}}$ are learned by backpropagation, and the gains also run the variance-matching rule above as a running estimate. This mirrors how Batch Normalization keeps running means. We will not pretrain from scratch, given compute. Instead we initialize $W$ and $g$ so that $M^{-1}$ approximates LayerNorm's per-token scaling, then fine-tune pretrained checkpoints on Kinetics-400 and Something-Something V2.

The local receptive field question enters through where the gain statistics are pooled. LayerNorm is blind to everything outside the current token. GainNorm's expectation is taken over all tokens in a frame or clip, so every token is rescaled using a summary of the whole scene. This is a cheap global-context channel that costs time linear in the number of tokens. It matters most in UniFormer's shallow, purely local stages. We will measure it by comparing per-token pooling against frame-level and clip-level pooling.

The hierarchy over time is explicit. At test time the gain is

$$g_t = g_{\text{src}} + \lambda_s\,\delta^{\text{slow}}_t + \lambda_f\,\delta^{\text{fast}}_t.$$

Here $\delta^{\text{fast}}$ is updated within a clip with a short time constant, $\delta^{\text{slow}}$ is updated across clips with a long time constant, and the shrinkage weights $\lambda_s, \lambda_f \in [0,1]$ pull both corrections back toward the stored source gains. This plays the role of NOTE's soft shrinkage between instance and source statistics. The difference is that here it acts on a dictionary rather than on per-channel moments, and on LayerNorm models rather than ResNets. No backbone weights change. Adaptation touches only the gain vectors, a footprint similar to the 0.02% of parameters NOTE reports for ResNet-18.

Baselines

The naive reference is the source model with no adaptation: the LayerNorm backbone run frozen on the corrupted stream. On UCF101 under corruption, ViTTA reports 51.35% accuracy for source-only TANet. That gives the scale of the gap any method must close.

The strongest existing competitor is ViTTA itself. It aligns test feature statistics to training statistics with exponential moving averages, works at batch size 1, and adds prediction consistency across resampled frame rates. It is the strongest because it already sidesteps the missing-running-statistics problem by storing feature moments externally, and it does so on video. It reports 78.20% on corrupted UCF101 with TANet. We run ViTTA on the same UniFormer and VideoMamba checkpoints, with the same corruption streams, the same number of adaptation steps per clip, and the same hyperparameter search budget. We count GainNorm as beating it if it matches or exceeds ViTTA's accuracy while updating only the gains rather than the full parameter vector. We also require it to clearly outperform ViTTA on the class-clumped streams described below, where ViTTA's moving averages should drift.

The LayerNorm counterpart of TENT, which adapts only the LayerNorm scale and shift parameters by minimizing prediction entropy, is the obvious parameter-efficient competitor. It gets the same steps and tuning budget. For reference, NOTE reports that BN-based TENT reaches 76.4% error on non-i.i.d. CIFAR10-C while NOTE reaches 21.1%. That gap is the failure mode we expect LayerNorm-TENT to inherit on clumped streams.

STAD tracks class prototypes in the penultimate layer with a probabilistic state-space filter. It is the strongest method that, like ours, avoids backpropagation through the feature extractor. We run it on the same features.

Two ablations isolate the key ingredient. The first is a complete diagonal variant ($K = d$, $W = I$), which is a per-channel running-variance normalizer with the same multi-timescale gains. The second is a single-timescale variant that keeps only $\delta^{\text{fast}}$. We also include an expensive upper bound: online full-covariance whitening with an exponentially averaged $d \times d$ covariance and an explicit inverse square root.

Why not just…?

Why not just adapt the LayerNorm affine parameters, as TENT does for Batch Normalization? Scale and shift parameters have no stored population anchor. On a stream dominated by one class, entropy minimization pushes them toward that class, which is the over-whitening collapse NOTE demonstrates for BN-based TENT. The clumped-stream experiment measures this directly. If LayerNorm-TENT holds up there as well as GainNorm does, the stored-anchor argument fails, and we would report that.

Why not just use ViTTA's external moment alignment, which already works on any architecture? This is the most serious threat. ViTTA stores means and variances outside the network, so it does not need a special normalization layer. Our bet rests on two differences. First, ViTTA matches moments but does not change how the network normalizes internally. Second, it has a single averaging rate with no anchor, which should drift on correlated streams. If ViTTA matches GainNorm on clumped streams, the contribution shrinks to parameter efficiency. The two-stream design in the experiments below separates these cases.

Why not just track per-channel variance, or full covariance, without an overcomplete dictionary? If corruptions mostly rescale channels, the diagonal ablation will match GainNorm and overcompleteness adds nothing. We will check this by sweeping $K \in \{d, 2d, 4d\}$; CRATE-α found that a 4× overcomplete dictionary was what let its sparse coding blocks scale. Full-covariance whitening should be at least as accurate. The question is whether GainNorm reaches most of its gain at a small fraction of the per-layer cost.

Proposed Experiments

The clean-accuracy check comes first. GainNorm-UniFormer-S, fine-tuned on Kinetics-400, should stay within one point of the reported 80.8% top-1 at 167 GFLOPs. On Something-Something V2 the targets are the reported 71.2% for UniFormer and 63.7% for VideoMamba, which we treat as reference points for the original checkpoints. If clean accuracy drops by more than a point, the layer is not a viable drop-in, and the adaptation results matter less.

The main experiment uses two kinds of stream on corrupted UCF101 and corrupted Kinetics-400, applying the ViTTA corruption suite. ViTTA reports no Kinetics-400 number in the file, so we rerun it ourselves. The i.i.d. stream shuffles videos. The clumped stream orders videos by a Dirichlet class distribution, following NOTE's protocol, and switches corruption type every few hundred clips. We report accuracy, recovery time after a corruption switch, and accuracy on the original domain after the stream returns to it, which measures forgetting.

The decisive result is a pattern across the two streams. GainNorm should roughly tie ViTTA on the i.i.d. stream and beat it, along with LayerNorm-TENT, on the clumped stream. The single-timescale ablation should lose the clumped-stream advantage, and the diagonal ablation should fall measurably below $K = 4d$. If any one of these fails, the result identifies which ingredient (the anchor, the timescales, or overcompleteness) is not doing the work.

A secondary check uses AffWild2 streams, where TempT reports F1 rising from 0.307 to 0.323 and decision flips falling from 0.15 to 0.043 per frame. We compare decision flicker under GainNorm adaptation against TempT's median-filtered targets, adapting a transformer backbone in both cases.