Spatial Correlation Sets How Far Each Normalization Layer Should Trust Test Statistics

When a vision model trained on clear daytime images meets fog, night, or rain, one of the cheapest fixes is to re-estimate the averages and spreads that its batch normalization layers use to rescale features. Batch normalization is the layer that standardizes each channel using a mean and variance, and it stores running averages of these from training. At test time, the question is how much to trust statistics computed from the few new images at hand versus the stored ones. Current methods answer with a single global mixing weight or momentum, or with a per-layer rule based on how different the new statistics look. This proposal argues that the missing variable is how reliable each test estimate is. That reliability depends heavily on how spatially correlated a layer's outputs are. A channel whose outputs are smooth across the image, such as one fed by a wide dilated convolution, gives far fewer independent observations per image than its pixel count suggests.

Multi-scale context modules are the cleanest setting to test this idea. These are blocks such as atrous spatial pyramid pooling (ASPP), which runs the same feature map through parallel "dilated" convolutions whose kernel taps are spread apart to see wider context without downsampling. Parallel branches with dilation rates of 1 and 18 see identical inputs and identical pixel counts, yet their outputs have very different spatial smoothness. We propose to estimate an effective sample size for every normalization layer from the spatial (and, for video, temporal) autocorrelation of its activations. That number then sets a shrinkage weight between source and test statistics, with one prior-strength constant shared across the whole network. The claim is that this rule, which needs no gradients or augmentations, recovers most of the gain of per-branch oracle tuning. It should also explain when uniform test-time normalization fails in dense prediction: at small batches, on temporally correlated driving streams, and mostly in the wide-context branches.

Core Claim

For a normalization layer with $N$ spatial positions per batch and an average spatial autocorrelation profile $\rho(k)$ at lag $k$, the effective number of independent observations is approximately $n_{\text{eff}} = N / (1 + 2\sum_{k \ge 1} \rho(k))$. This is the standard variance-inflation correction for correlated samples. We estimate each test-time channel mean and variance as a Gaussian posterior mean,

$$\hat{\mu} = \frac{n_{\text{eff}}\,\mu_{\text{test}} + \kappa\,\mu_{\text{src}}}{n_{\text{eff}} + \kappa},$$

where $\mu_{\text{test}}$ is the current batch statistic, $\mu_{\text{src}}$ is the stored running statistic, and $\kappa$ is a single prior-strength constant for the whole network (variances are treated the same way). The rule makes three falsifiable predictions. First, wide-dilation branches get systematically lower $n_{\text{eff}}$ and should trust test statistics less. Second, the per-layer weight found by oracle search on labeled target data should track $n_{\text{eff}}/(n_{\text{eff}}+\kappa)$. Third, the advantage over a uniform rule should vanish with large, shuffled, i.i.d. batches and grow at batch size one and on ordered video streams.

Relation to Prior Work

Several methods motivate the proposal by showing that naive re-estimation of normalization statistics breaks down. Tent adapts normalization scale and shift by entropy minimization while re-estimating batch statistics, and it collapses at batch size one. MixNorm and TEMA document that collapse and fix it with fixed-weight global/local blending and with momentum plus KL-divergence-based per-layer rectification. NOTE shows that temporally correlated streams over-whiten features and responds with instance-aware normalization that shrinks toward instance statistics. DUA updates running statistics without gradients using a decaying momentum that is uniform across layers.

These methods share the shrinkage idea, and we keep it. None of them sets the shrinkage by the statistical reliability of each layer's estimate. TEMA's rectification is closest, and it scores divergence between source and target, not estimation noise. A large divergence estimated from few effective samples is exactly the case in which TEMA's rule and ours disagree, and that is where we expect our rule to win.

Layer-selection methods constrain the design. GALA masks whole-layer gradient updates by cosine alignment. DAT picks parameter subsets from pixel uncertainty, and CoTTA updates all weights under a mean teacher with stochastic restoration. They show that indiscriminate updating causes forgetting, but none of them looks inside a multi-scale block. L-TTA and MGIPT handle scale through wavelets at the network stem or through input prompts, and they leave deep context branches frozen.

On the architecture side, RFB Net builds batch-normalized parallel branches whose dilation grows with kernel size, and it never studies them under shift. ASPP-style work on gridding, such as Augmented ASPP, notes that sparse dilated sampling changes spatial structure. That observation is the origin of the correlation differences we exploit.

One paper limits our scope. LN-TTA argues that layer normalization, which standardizes each sample on its own, should not be re-standardized across batches. The proposal therefore targets batch- and group-normalized dense heads, which remain standard in dilated segmentation and detection models. We do not claim it transfers to layer-normalized Transformers.

Research Program

The first stage is diagnostic. We take an ASPP segmentation model trained on Cityscapes and run it on ACDC fog, night, rain, and snow. For every normalization layer we measure three things: the shift between stored and target statistics, the spatial autocorrelation of activations, and the resulting $n_{\text{eff}}$ at batch sizes 1, 4, and 16. Branch-level comparisons inside ASPP hold input and pixel count fixed, so any difference in $n_{\text{eff}}$ comes from dilation rate alone. This stage answers the open question of whether normalization drift and estimation noise differ across receptive-field scales, which no method in the file has measured.

The second stage builds the estimator. We compute $n_{\text{eff}}$ online from a cheap estimate of channel-averaged autocorrelation at a few lags. For video we multiply in a temporal factor estimated from frame-to-frame correlation of per-channel means, which targets NOTE's over-whitening regime. The estimator then applies the shrinkage rule with one $\kappa$ tuned once on a held-out corruption and frozen for all targets. The method is forward-only, needs no augmentations, and adds one autocorrelation computation per normalization layer. Optionally, the corrected statistics can serve as the base for Tent-style affine updates, to test whether better statistics also stabilize gradient-based adaptation.

The third stage tests generality. We apply the rule to RFB Net for foggy detection and to plain ResNet classifiers. In a classifier, spatial correlation changes with depth rather than across parallel branches, so the rule should still produce layer-dependent shrinkage.

Baselines

The naive reference points are the unadapted source model and plain test-time batch normalization, which recomputes statistics from each test batch with full trust. Both run with the same checkpoint and batch sizes as our method. Beating them is the minimum bar, and we expect plain re-estimation to fall below the source model at batch size one.

The strongest statistics-only competitor is TEMA's momentum with KL-based layer-wise rectification (TEMA). It already blends source and target statistics per layer, so it is the direct test of whether reliability beats divergence as the blending signal. On CIFAR-10-C at batch size one TEMA reports 21.05% error, and MixNorm reports 21.94%. We rerun both on our segmentation models and give each the same hyperparameter search budget, measured in held-out tuning runs, that we give $\kappa$. Beating TEMA means lower error or higher mean intersection-over-union (mIoU) at batch sizes 1 and 4 on ordered streams, with a gap larger than seed variance over at least three seeds. NOTE's instance-aware normalization is the second statistics-level competitor for the correlated-stream setting. It reports 21.1% error on non-i.i.d. CIFAR-10-C, against 76.4% for Tent, and it is run under the same stream orderings as our method.

The strongest continual-segmentation competitors are DAT and CoTTA. On Cityscapes-to-ACDC, DAT reports 60.8% mIoU, CoTTA 58.6%, and the source model 56.7%, while Tent degrades to 55.0% over rounds. Those numbers come from those papers' own backbones, which the file does not specify. We therefore rerun DAT and CoTTA on our ASPP model and compare on equal footing. We report both accuracy and wall-clock and memory cost, because both methods use augmentation-heavy teachers. Matching DAT's reproduced mIoU at a fraction of its compute would count as success for a gradient-free method. Exceeding it when our rule is combined with Tent-style affine updates would be the stronger result.

Two ablations isolate the key ingredient. The first replaces $n_{\text{eff}}$ with the raw pixel count $N$, which collapses the rule to count-based blending close to MixNorm. The second is a per-branch oracle that grid-searches each layer's mixing weight on labeled target data. The first ablation shows whether the correlation correction matters at all. The second shows the ceiling our rule should approach. Our target is to recover at least most of the oracle's gain over the uniform rule.

Why not just…?

Why not just tune a separate momentum for each ASPP branch? This is the most serious threat, and the per-branch oracle ablation measures it directly. Per-branch tuning uses target labels and multiplies the number of hyperparameters by the number of normalization layers, so it is not deployable. The real question is whether a label-free rule can match it. If the oracle's chosen weights do not track $n_{\text{eff}}$, the core claim fails and we will report that.

Why not just use TEMA's KL rectification, which is already per-layer? TEMA measures how far target statistics sit from source statistics. A wide branch under dense fog can show a large divergence that is mostly estimation noise from few effective samples, and TEMA would then trust that noise more. The head-to-head test at batch size one on ordered ACDC sequences is built to expose exactly this disagreement. If TEMA matches us there, reliability adds nothing beyond divergence, and the proposal reduces to a diagnostic study.

Why not just set shrinkage by dilation rate, a hand-made heuristic? Dilation is a proxy for spatial correlation, and it may capture much of the effect inside ASPP. It cannot explain differences across depth in plain backbones, and it cannot capture temporal correlation in video. We include a dilation-proportional rule as an extra baseline. If it matches our rule on ASPP but fails on RFB Net or ResNet classifiers, that supports measuring correlation over guessing it from architecture.

Why not just adapt at the stem or the input, as L-TTA and MGIPT do? These methods are cheap. Our diagnostic stage checks whether drift in deep context branches remains after stem-level correction. We run L-TTA and then measure residual statistic shift in the ASPP branches. If that residual is small, deep adaptation is unnecessary and we will say so.

Proposed Experiments and Decisive Checks

The main benchmark is Cityscapes-to-ACDC in the continual protocol of CoTTA and DAT, run at batch sizes 1, 4, and 16. Each size is tested in two versions, with frames shuffled and with frames kept in temporal order. Cityscapes-C is the second segmentation benchmark. L-TTA reports a 4.6% mIoU gain over EcoTTA there, and no absolute mIoU for our intended backbone was found on Lacuna, so we report our own reruns. For detection we apply the method to RFB Net on KITTI-Fog. DUA reports YOLOv3 improving from 30.9% to 51.4% mAP there, and no reported result was found on Lacuna for RFB Net on KITTI-Fog. CIFAR-10-C at batch size one and under NOTE's non-i.i.d. ordering serves as a classification sanity check. Metrics are mIoU, mAP, or error; per-round degradation; per-frame latency; and memory.

Three outcomes decide the proposal. First, the correlation between oracle per-layer mixing weights and $n_{\text{eff}}/(n_{\text{eff}}+\kappa)$ must be clearly positive across branches and corruptions. Second, the gain over the raw-count ablation must concentrate at small batches and ordered streams, and shrink toward zero with large shuffled batches, exactly as the variance argument predicts. Third, per-branch measurement must show whether wide branches carry disproportionate drift. That measurement is informative even if the adaptation rule underperforms, because no work in the file has characterized it.