Noise-Calibrated Filtered Batch Normalization for Test-Time Shift

Batch normalization rescales each channel of a network's activations using a mean and variance. During training those statistics come from the current mini-batch. At deployment they are frozen to averages collected on training data. When the test data differs from the training data, the frozen statistics are wrong. Recomputing them on the test batch fixes that, but at small batch sizes, or when a batch happens to contain only a few classes, the recomputed statistics are dominated by sampling noise and accuracy collapses. Current methods mix the two estimates using a weight that is either fixed in advance or switched by a hand-set threshold.

Numerical PDE solvers faced the same dilemma decades ago. They had an accurate operator that is unreliable near shocks and a crude operator that is always safe. The cleanest resolution, the filtered scheme of Froese and Oberman, blends the two through a continuous function of their difference, scaled by the known error size of the schemes, rather than by a free tuning constant. We propose the analogous rule for normalization. For each layer and channel, a test statistic is trusted only to the extent that its gap from the training statistic exceeds what sampling noise would produce at the current batch size and class mixture. That noise level is measured once, offline, by resampling source data.

Core Claim

A single quantity can replace the learned or hand-tuned blending weights that current test-time normalization methods depend on. That quantity is a per-layer, per-channel, batch-size-dependent "noise envelope" of normalization statistics under the source distribution, computed under realistic label skew and with depth compounding included. Feeding the observed source-to-test gap, divided by this envelope, into a continuous filter does three things. It follows genuine covariate shift where shift exceeds noise. It falls back to source statistics where the gap is indistinguishable from small-batch or label-skew noise. It produces the layer- and channel-selective behavior that other methods impose by design: early layers adapt and class-sensitive deep channels stay frozen, but here this emerges from measurement.

The claim is falsifiable in a specific way. A version of the method whose weight depends only on batch size, and not on the observed gap, should do measurably worse. So should a version calibrated on balanced batches only.

What the Two Directions Contribute

From adaptive-order numerics we take the structure of the filter, not a physical analogy. Froese and Oberman blend a second-order operator with a monotone first-order one through $F_M + \epsilon\,\phi((F_A-F_M)/\epsilon)$. The threshold $\epsilon$ is tied to the schemes' formal accuracy, and the output never strays more than about $\epsilon$ from the safe operator. Shock-capturing methods such as the subcell limiters in FLEXI and the admissibility-preserving LWFR limiter also show that the sensor's sensitivity is the weak point: a sensor that is too eager clips smooth peaks, and one that is too lax lets oscillations through. That literature lists hand-tuned indicator thresholds as an open problem, which motivates us to calibrate rather than tune.

The direction of trust is inverted relative to the PDE case, and we say so plainly. In Froese–Oberman, a large disagreement indicts the accurate scheme. Here, the estimator whose error we can bound is the noisy test statistic, so a large disagreement, beyond the noise envelope, indicts the frozen source statistic. What carries over is the principle that a continuous filter on the difference between two estimators, normalized by the measured error level of the noisier one, removes the free threshold.

The normalization literature supplies the problem and the competitors.

  • Plain re-estimation. AdaBN and covariate shift adaptation show that re-estimating statistics on test data is a strong, training-free fix when batches are large.
  • Static learned weights. TTN learns a static channel-wise mixing weight on augmented source data. It finds shallow layers want test statistics and deep layers want source ones.
  • Temporal pooling with a hard fallback. TEMA pools test statistics over time and reverts a layer to source statistics when a KL divergence crosses a threshold.
  • Label-shift channel freezing. Hybrid-TTN freezes class-sensitive channels using Wasserstein profiling combined with a fixed linear schedule over depth.
  • Depth compounding. RBN argues the opposite depth story: estimation error compounds across layers, so deep BN layers should be replaced by group normalization.
  • Shrinkage. JS-BN applies James–Stein shrinkage to batch statistics under noise and adversarial attack.

Our method differs from all of these. Its weight is recomputed for every batch from the observed gap. It has no learned mixing parameter, no fixed depth schedule, and no free divergence threshold. Its calibration explicitly includes label-skewed batches and the propagation of errors through the earlier, already-filtered layers.

Research Program

The method has three parts.

Envelope estimation. This is done offline on held-out source data. For each layer $l$, channel $c$, and batch size $N$ on a grid from 1 to 256, we draw many source batches. Their label mixtures come from a Dirichlet prior that ranges from balanced to single-class. Each batch is passed through the network with the filter already active in shallower layers, so depth compounding is part of the calibration. We record the spread of the gap between batch statistics and running statistics. Concretely, $\varepsilon^{\mu}_{l,c}(N)$ is the $q$-quantile of $|\mu^{B}_{l,c} - \mu^{s}_{l,c}|$ over resampled source batches $B$ of size $N$. Here $\mu^{s}$ is the stored running mean, and $q$ is the single global confidence level. The variance envelope $\varepsilon^{\sigma}_{l,c}(N)$ is defined the same way on $\log(\sigma^2_B/\sigma^2_s)$. The result is a small lookup table, comparable in size to the BN parameters themselves.

The test-time filter. Given the test statistic $\mu^{t}$, which is either the current batch or a TEMA stream with an effective sample size, the filter computes a normalized gap and blends:

$$z_{l,c} = \frac{|\mu^{t}_{l,c}-\mu^{s}_{l,c}|}{\varepsilon^{\mu}_{l,c}(N)}, \qquad \hat\mu_{l,c} = \mu^{s}_{l,c} + w(z_{l,c})\,(\mu^{t}_{l,c}-\mu^{s}_{l,c}).$$

The weight $w(z)=\min(1,\max(0,z-1))$ is zero inside the envelope and reaches full trust at twice the envelope. The variance is filtered in log space in the same way. The filter is continuous, which matters for streaming stability. It is applied layer by layer in the ordinary forward pass, at no optimization cost.

Analysis. The analysis has two goals. The first is to test whether the per-layer average of $w$ reproduces TTN's shallow-adapt, deep-freeze pattern, and whether the channels with wide label-skew envelopes coincide with Hybrid-TTN's Wasserstein-sensitive channels. The second is to resolve the depth contradiction between TTN and RBN. We will measure envelope growth with depth under sequential calibration. If envelopes widen with depth, deep layers freeze because compounded noise exceeds shift, which is consistent with both papers. If they do not, RBN's exponential compounding claim is not reproduced in pretrained BN networks at test time.

Evaluation setting.

  • Benchmarks: CIFAR-10-C, CIFAR-100-C, and ImageNet-C at severity 5 and across severities.
  • Batch sizes: a sweep from 200 down to 1.
  • Label shift: label-shifted CIFAR-10 test streams, including single-class batches, following Hybrid-TTN.
  • Segmentation: Cityscapes→BDD-100K transfer.
  • Streaming: gradual-severity sequences, scored for prediction flip rate in the spirit of ImageNet-P.
  • Backbones: standard pretrained ResNets (ResNet-26/WRN on CIFAR, ResNet-50 on ImageNet). Weights are never updated.

Like TTN and Hybrid-TTN, the method needs offline access to source data and source labels for calibration. We state this restriction rather than claim source-free operation.

Baselines

Naive references. These mark the two ends of the trade-off.

  • Frozen source statistics (no adaptation). Hendrycks and Dietterich report mCE 76.7 for ResNet-50 on ImageNet-C.
  • Full test-batch re-estimation (TBN/AdaBN). Schneider et al. report that it reduces ResNet-50 mCE to 62.2 with large batches. TTN and TEMA report about 90% CIFAR-10-C error at batch size 1.

Any method worth proposing must be no worse than the better of these two at every batch size. That condition is our first pass/fail line.

Static learned mixing. TTN is the strongest competitor among methods that learn a fixed mixing weight from source data. It reports about 18% CIFAR-10-C error at batch size 1, 78.48% average ImageNet-C error for ResNet-50 versus 86.40% for TBN, and 47.40 versus 43.12 mIoU on Cityscapes→BDD-100K.

Dynamic fallback. TEMA is the strongest dynamic competitor, reporting 21.05% CIFAR-10-C error at batch size 1 and 20.20% at batch size 200.

Label shift. Hybrid-TTN is the strongest label-shift method. It keeps about 87% CIFAR-10 accuracy on single-class batches, where TTN falls to about 20%.

We will rerun all of these from released code where available, on identical pretrained checkpoints, corruption streams, and batch orders. Each competitor gets the same held-out source data for its calibration step. Its free hyperparameters (TEMA momentum and KL threshold, TTN post-training schedule) get the same tuning budget we give our single quantile $q$, tuned on a disjoint corruption split.

Beating these baselines means the following. We must match or beat TTN at batch size 1 on CIFAR-10-C within run-to-run noise. We must beat both TTN and TEMA in the worst case over the batch-size sweep, measured as maximum error across $N$. We must stay within a few points of Hybrid-TTN on single-class batches while beating it on balanced corrupted streams.

Shrinkage baseline. This is a heteroscedastic James–Stein estimator that shrinks test statistics toward source statistics. Its noise variances are taken from our own envelopes, extending JS-BN, which shrinks toward a fixed target rather than toward source statistics.

Ablations. Two ablations isolate the key ingredients.

  • Batch-size-only weight: the weight depends on $N$ alone, not on the observed gap.
  • Balanced-only calibration: the envelope is calibrated without label skew.

Why not just…?

Why not just replace batch normalization with group or layer normalization? That removes batch dependence, but it requires retraining. It also discards the property that BN statistics carry domain style, which AdaBN and Schneider et al. exploit for training-free transfer. RBN itself finds that a hybrid, keeping BN early and using GN deeper, beats uniform replacement. We include RBN-style hard replacement of deep layers as a comparison. If our envelopes freeze the same deep layers RBN replaces, that supports measured selection over fixed architecture.

Why not just shrink test statistics toward source statistics with James–Stein? This is the real threat. Our filter is a per-coordinate cousin of positive-part shrinkage, and much of any gain may come from the calibrated envelopes rather than the filter's shape. We therefore give the shrinkage baseline the same envelopes. If it matches us, we will report that the contribution is the label-skew- and depth-aware calibration, not the PDE-style filter. We expect the per-channel filter to win specifically on label-shifted batches, where a pooled shrinkage factor cannot freeze class-sensitive channels while adapting noise-sensitive ones.

Why not just tune TTN's static weights or TEMA's threshold harder? TTN's weights are fixed before deployment, so they cannot respond to how severe the shift actually is. TEMA's switch is binary, so it can flip a layer between regimes across consecutive frames. The batch-size sweep and the gradual-severity stream test both weaknesses directly, under equal tuning budgets.

Why not just train robustly with augmentation? Schneider et al. show the two are additive: DeepAugment+AugMix improves from 53.6 to 45.4 mCE once BN adaptation is added. So we will also run our filter on an augmentation-trained backbone rather than treat robust training as a rival.

Decisive Experiments

First check: never worse than either extreme. On the CIFAR-10-C batch-size sweep, the filtered method's error must stay at or below the minimum of source-only and TBN at every $N$, and must be competitive with TTN at $N=1$. Failure here sinks the proposal.

Second check: the gap-dependent weight matters. The batch-size-only ablation must lose clearly on mixed-severity streams. There, the same $N$ calls for adaptation at severity 5 and none at severity 1. If it does not lose, the observed gap adds nothing and the method reduces to a lookup table.

Third check: label-skew calibration matters. On single-class CIFAR-10 batches, balanced-only calibration should collapse toward TTN's roughly 20% accuracy, while full calibration should approach Hybrid-TTN's roughly 87% without its depth schedule. Beyond accuracy, the channels our envelopes freeze should overlap substantially with Hybrid-TTN's Wasserstein-sensitive set. We will measure this overlap with rank correlation.

Fourth check: streaming stability. On gradual-severity streams, the continuous filter should yield a lower prediction flip rate than TEMA's hard fallback at equal accuracy.

Fifth check: scale and dense prediction. ImageNet-C with ResNet-50 at small batch sizes, and Cityscapes→BDD-100K, test whether the gains survive scale and dense prediction. The targets are the TTN figures of 78.48% average error and 47.40 mIoU.