When a trained image classifier meets shifted inputs at deployment, such as blurred, noisy, or differently scanned images, a common fix is to re-estimate the per-channel mean and variance that its normalization layers use, computed from the incoming test images. This re-estimation breaks in two familiar situations. Batches can be tiny. Batches can also contain only one or two classes, because a deployed camera or a single hospital sees correlated streams. In both cases the test-batch statistics mix two things: the real domain shift, which the model should absorb, and fluctuations caused by which classes happen to be in the batch, which it should ignore.
We propose to separate these two signals with the oldest tool for the job, analysis of variance (ANOVA). On the source training data, ANOVA splits each channel's variance into a between-class part and a within-class part. A channel whose activity is mostly between-class carries class identity, so a shift in its batch mean is likely caused by class composition rather than by the domain. We feed this per-channel split into a small online estimator whose update rate depends on its own uncertainty, modeled on the normative account of short-term synaptic depression. In that account, a depleting resource variable tracks how uncertain the synapse's estimate is and sets how fast the synapse adapts. The result is a normalization layer in which class-carrying channels adapt slowly and class-agnostic channels adapt quickly. It requires no batch statistics beyond what arrives and no hand-set depth schedule. We then use functional ANOVA, applied to test error across a factorial grid of shift conditions, to check whether the method removes the specific interaction that makes current methods fail.
Core Claim
Test-time normalization fails under small or class-imbalanced batches because, in class-discriminative channels, the batch mean is a noisy observation of domain shift. The noise level of that observation can be predicted in advance from the source data's between-class variance fraction. An estimator that uses this predicted noise, with a per-channel and per-time-step adaptive rate in the style of a depressing synapse, should match the strongest channel-selective method on label-shifted streams while staying stable at batch size 1. It should also replace that method's heuristic channel scores and depth decay with a quantity computed once from training data. The evaluation claim is sharper still: under batch normalization the error variance across conditions should be dominated by a batch-size × class-concentration interaction, and our method should shrink that interaction term, not merely the average error.
Where This Sits
Two test-time adaptation papers motivate the gap.
Unraveling Batch Normalization for Realistic Test-Time Adaptation shows that small batches fail because they lack class diversity, not because they are small. Its fix is an exponential moving average of test statistics (TEMA) with a fixed momentum. We keep the moving-average idea but make the rate per-channel and uncertainty-driven.
Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation introduces Hybrid-TTN. It freezes deeper, class-carrying channels using 2-Wasserstein channel scores and a linear decay over depth. It is the closest prior work. We share its intuition that discriminative channels should not be re-estimated. We differ in deriving which channels to protect, and by how much, from a variance partition of source data. We also replace the binary-ish freezing with a continuous online gain that responds to accumulated evidence.
The method itself comes from the cellular-adaptation literature. Know Thy Neighbour: A Normative Theory of Synaptic Depression recasts the Tsodyks–Markram depressing synapse as a Bayesian estimator. In that view, the postsynaptic potential tracks the estimated mean and the resource variable tracks its uncertainty, so the effective learning rate is uncertainty-gated. That paper also shows that the depressing synapse beats a static one in reconstruction error. We borrow this estimator form. We are honest that in its simplest version it is close to a scalar Kalman filter, and the experiments test whether the biologically motivated details matter.
Adaptive whitening with fast gain modulation and slow synaptic plasticity supplies the two-timescale picture: fast gains absorb context shifts while a slow basis stays fixed. Our split of channels into fast-adapting and slow-adapting is a discrete analogue of that picture. The physiology of Multiple Timescales of Adaptation in a Neural Code and Shifts in Coding Properties and Maintenance of Information Transmission during Adaptation in Barrel Cortex supports it as well. Those papers report gain rescaling within 100–200 ms and about 280 ms, while slower signals settle over seconds.
Divisive Feature Normalization Improves Image Recognition Performance in AlexNet constrains the design. It shows that divisive normalization helps as a fixed trained layer, but it does not adapt at test time. We include it as a batch-independent alternative rather than build on it.
The evaluation lens comes from What causes the test error? Going beyond bias-variance via ANOVA. That paper shows that the double-descent peak comes from a sample × initialization interaction rather than from either factor alone. Its analysis is exact but limited to lazy two-layer models. We use its decomposition empirically, without random-matrix asymptotics, on adaptive deep networks.
Method
For each normalized channel $c$, we compute on source data the between-class fraction $$\eta_c^2 = \frac{SS_{\text{between},c}}{SS_{\text{total},c}}.$$ Here $SS_{\text{between},c}$ is the variation of the class-conditional mean activations around the grand mean, and $SS_{\text{total},c}$ is total activation variation. From the same pass we store the within-class variance $\sigma^2_{w,c}$ and the variance of the class means $\sigma^2_{b,c}$.
At test time, each channel keeps an estimate $\hat\mu_c$ of the current domain's mean, together with an uncertainty $P_c$. The batch-level variance estimate gets an analogous treatment. A batch of size $B$ containing $k$ distinct predicted classes is treated as an observation $\bar a_c$ of $\hat\mu_c$. Its expected noise is $$R_c \approx \frac{\sigma^2_{w,c}}{B} + \frac{\sigma^2_{b,c}}{k}.$$ The first term is ordinary sampling noise. The second is class-composition noise, which does not shrink with batch size when classes are few. This is exactly the diversity gap identified by Su et al. The update is $\hat\mu_c \leftarrow \hat\mu_c + K_c(\bar a_c - \hat\mu_c)$ with gain $K_c = P_c/(P_c + R_c)$. The uncertainty decreases after each update and grows by a process-noise term $Q$ between batches, which allows for continued domain drift. $Q$ is the only global hyperparameter.
Under this rule, high-$\eta^2$ channels in single-class streams receive near-zero gain and stay close to source statistics. Low-$\eta^2$ channels, typically early and texture-like, track the corruption quickly. The synaptic-depression variant replaces the additive uncertainty dynamics with a multiplicative resource that depletes on large prediction errors and recovers at a fixed rate, following the Tsodyks–Markram form in Know Thy Neighbour.
The number of distinct classes $k$ must be estimated from the model's own predictions, which may be wrong under shift. An ablation therefore uses the oracle $k$ to measure how much this matters.
Research Program
The work has three parts.
The first is diagnostic. We build a factorial grid of conditions crossing five factors: corruption type and severity, batch size, label-stream concentration (a Dirichlet parameter over classes, from i.i.d. to single-class), stream order, and source seed. For each adaptation method we fit a functional ANOVA to test error and report main effects and two- and three-way interaction variance components. The diagnostic question is whether the batch-size × concentration interaction dominates error variance under batch-norm adaptation, in the way the sample × initialization interaction dominates in Lin and Dobriban's analysis.
The second part builds the variance-partitioned estimator and its two variants. The Kalman-style version uses additive uncertainty; the depression-style version uses a multiplicative resource. Both are dropped into the normalization layers of the same source models.
The third part checks whether $\eta_c^2$ actually predicts harm. We correlate each channel's source between-class fraction with the accuracy lost when that channel alone is re-estimated on single-class batches. If this correlation is weak, the premise fails, and we would learn that early.
Baselines
The naive reference point is the unadapted source model, evaluated on identical streams. Next is standard test-time batch normalization (TTN), which recomputes statistics from each batch. According to Vianna et al., its CIFAR-10 accuracy falls from about 95% to about 20% under extreme single-class imbalance, so any method must clearly clear that floor.
The fixed-momentum baseline is TEMA from Su et al. Its reported CIFAR-10-C error rises only from 20.20% at batch size 200 to 21.05% at batch size 1, while TENT reaches 90%. This makes it the strongest competitor on the batch-size axis. We will run it with the authors' recipe and also with per-layer momentum tuned under the same validation budget we give our $Q$.
The strongest label-shift competitor is Hybrid-TTN. Vianna et al. report about 87% accuracy under extreme single-class imbalance on CIFAR-10. They also report median rank 1.0 on CIFAR-10-C, 2.3 on ImageNet-1K-C, and 1.5 on the liver ultrasound data. No reported error number on ImageNet-1K-C was found on Lacuna for it. We will rerun Hybrid-TTN ourselves on all streams with its released configuration and an equal hyperparameter search.
All methods share source checkpoints, test streams, label-stream seeds, and a fixed validation budget drawn from held-out corruptions not used in final reporting. Beating Hybrid-TTN means two things on CIFAR-10-C and ImageNet-1K-C, judged by paired comparisons over stream seeds:
- at least matching its single-class accuracy while improving on it at batch size 1 under i.i.d. labels;
- a significantly smaller batch-size × concentration variance component in the functional ANOVA.
Two ablations isolate the key ingredient. The first gives every channel the same observation noise, removing the $\eta^2$ term. The second replaces $\eta^2$ with Hybrid-TTN's Wasserstein channel scores inside our estimator. If either matches the full method, the variance partition is not doing the work.
Why not just…?
Why not just use a batch-independent normalization such as layer, group, or divisive normalization, and skip test statistics altogether? Batch-independent normalization does avoid the diversity gap. Divisive Feature Normalization Improves Image Recognition Performance in AlexNet shows that divisive normalization combined with batch norm reached 61.33% ImageNet top-1 versus 59.58% for batch norm alone, and it improved robustness to Gaussian noise. However, these layers have no mechanism for absorbing a domain shift after training, and they require retraining the source model. We will train a divisively normalized source model with matched compute and evaluate it unadapted on the same streams. If it matches adapted methods under corruption, that is a real result against all test-time normalization, including ours, and we will report it.
Why not just tune TEMA's momentum per layer? This is the most serious threat. Depth-wise momentum could recover much of the benefit if class-carrying channels concentrate in deep layers. The tuned-momentum baseline and the equal-noise ablation settle the question. Our prediction is that $\eta^2$ varies substantially within layers, so a per-layer rate cannot match a per-channel one on single-class streams. If the gap is negligible, the honest conclusion is that a tuned moving average suffices, and the contribution reduces to the ANOVA diagnostic.
Why not just report accuracy instead of a functional ANOVA? Average accuracy can improve while the failure mode stays intact, for example by gaining on i.i.d. streams and losing on concentrated ones. Lin and Dobriban show in What causes the test error? Going beyond bias-variance via ANOVA that aggregate error hid an interaction-driven mechanism. The variance components tell us whether a method fixed the interaction or merely shifted the mean. We report both, so a reader who distrusts the decomposition loses nothing.
Why invoke synaptic depression rather than a plain Kalman filter? This may be unnecessary. The depression-style variant competes directly against the Kalman-style variant. If they tie, we credit the biology as the source of the uncertainty-gated design and not as a performance gain.
Decisive Checks
The experiments use CIFAR-10-C and ImageNet-1K-C under the five-factor grid, plus the liver ultrasound cross-hospital set if it is accessible under the protocol of Vianna et al. Metrics are top-1 error, worst-stream error, and the functional ANOVA variance components with confidence intervals obtained by resampling stream seeds.
Three outcomes decide the proposal. First, the source between-class fraction $\eta_c^2$ must correlate with per-channel re-estimation harm; otherwise the core premise is false. Second, under batch normalization the batch-size × concentration component must account for a large share of error variance, and our estimator must reduce that share relative to TEMA and Hybrid-TTN. Third, the equal-noise ablation must be clearly worse than the full method. If the third check fails, the variance partition is decorative and we drop it from the claim.
References
- Unraveling Batch Normalization for Realistic Test-Time Adaptation
- Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation
- Know Thy Neighbour: A Normative Theory of Synaptic Depression
- Adaptive whitening with fast gain modulation and slow synaptic plasticity
- Multiple Timescales of Adaptation in a Neural Code
- Shifts in Coding Properties and Maintenance of Information Transmission during Adaptation in Barrel Cortex
- Divisive Feature Normalization Improves Image Recognition Performance in AlexNet
- What Causes the Test Error? Going Beyond Bias-Variance via ANOVA