Auxiliary Heads as Test-Time Probes for Normalization Repair

Batch normalization rescales each layer's activations using a mean and variance. During training these come from the current batch; at deployment they are frozen averages from the training set. When test images look different (fog, night, a different scanner), the frozen averages no longer match, features get mis-scaled, and accuracy drops. Re-estimating the statistics from test data fixes much of this for classification, but in semantic segmentation (labeling every pixel) it often makes things worse. Choosing how much to trust test statistics at each depth of the network is also hard without labels. Separately, many segmentation networks are trained with deep supervision: small prediction heads attached to intermediate layers, each producing its own coarse segmentation map. These heads are almost always thrown away after training.

This note proposes keeping those heads and using them at test time as cheap, label-free probes. A head attached after stage $s$ can only be affected by the normalization statistics of stages $1$ through $s$. Watching how each head's prediction changes as we vary the statistics at each stage therefore tells us where in the network the mismatch is doing damage. The final output alone cannot tell us this. We use the heads' cross-scale agreement to choose, stage by stage and without source data or gradients, how to blend frozen source statistics with test statistics.

Core Claim

The claim is that intermediate heads trained with deep supervision provide a depth-localized signal for normalization repair that neither statistic-distance heuristics nor final-output confidence provide. Concretely, for each encoder stage $s$ we blend statistics as in TTN,

$$\hat{\mu}_s = \alpha_s \mu^{test}_s + (1-\alpha_s)\mu^{src}_s, \qquad \hat{\sigma}^2_s = \alpha_s \sigma^{2,test}_s + (1-\alpha_s)\sigma^{2,src}_s,$$

where $\mu^{src}_s, \sigma^{2,src}_s$ are the stored training statistics, $\mu^{test}_s, \sigma^{2,test}_s$ are a moving average over the test stream, and $\alpha_s \in [0,1]$ is the per-stage mixing weight. TTN tunes $\alpha$ offline with labeled source data and then freezes it. We instead choose $\alpha_s$ online by a forward-only search that maximizes agreement among the heads.

The scoring function has two parts. Let $p_k$ be the pixelwise class distribution from head $k$ (downsampled to a common resolution), and let $\bar{p}_K$ be the image-averaged class distribution of the final head. The score is

$$J(\alpha) = -\sum_{k} \mathrm{KL}\big(p_{k+1} \,\|\, p_k\big) \;-\; \lambda \,\max\big(0,\; \tau - H(\bar{p}_K)\big),$$

where the sum runs over adjacent heads. The second term penalizes collapse onto a few classes: $H$ is entropy, $\tau$ is set from the class-frequency entropy of the training labels (a stored scalar, not source data), and $\lambda$ weights the penalty. Because head $k$ depends only on $\alpha_{1..s(k)}$, we search coordinate-wise from shallow to deep over a small grid. Cost is about grid-size × number-of-stages forward passes, and we amortize it by re-searching only when the test statistics drift.

This also gives a diagnostic payoff. The literature disagrees on which depth suffers most. TTN finds shallow layers are the most shift-sensitive. Reducing Divergence in Batch Normalization for Domain Adaptation, which defines an Estimation Shift Magnitude and fixes it by swapping deep-stage BN for group normalization, finds error compounds into deep layers. The per-stage $\alpha$ profile our probes select, compared against oracle profiles chosen with labels, measures directly which picture holds for dense prediction under each shift type.

Research Program

The work proceeds in three parts.

First, an oracle study establishes whether the signal exists at all. On labeled target data we sweep per-stage $\alpha$ grids and record true mean intersection-over-union (mIoU) alongside $J$, alongside final-head entropy, and alongside the per-layer source/test KL divergence used by TEMA (Unraveling Batch Normalization for Realistic Test-Time Adaptation). The question is which unsupervised score rank-correlates best with mIoU, and whether head agreement localizes the stage where the oracle wants a change.

Second, we build the online method on two deeply supervised segmenters. The first is the multi-exit HRNet-W48 from liuzhuang13/anytime, whose exits already exist and are trained with intermediate losses. The second is a ResNet DeepLab-style model we retrain with decoder-level deep supervision in the style of Deeply-Supervised Nets. Running both shows the result does not depend on one architecture.

Third, we stress the method under small batches (batch size 1, class-imbalanced streams) and continual shifts. The test-statistic estimate there follows TEMA's moving average, and our contribution is only the choice of $\alpha$.

One known weakness shapes the design. Shallow heads are weak: the first HRNet exit reaches only 34.3 mIoU on Cityscapes versus 80.4 for the fourth, per the anytime resource. We therefore score agreement between adjacent heads rather than against the final head, and we report whether dropping the weakest head changes results.

Baselines

The naive reference is the unadapted source model with frozen statistics. Next to it sits full test-statistic re-estimation, i.e. AdaBN (Revisiting Batch Normalization For Practical Domain Adaptation) and covariate shift adaptation, equivalent to $\alpha_s=1$ everywhere. We expect this to underperform the source model on segmentation, consistent with A Critical Look at Classic Test-Time Adaptation Methods in Semantic Segmentation. That paper reports TENT lowering average mIoU on ACDC/Cityscapes-fog/rain from 61.0% to 58.8%.

The strongest source-free competitor on this exact question is TEMA with its layer-wise rectification. It is unsupervised, per-layer and gradient-free, and it chooses blends from source/test KL divergence. It differs from our method only in the signal used, which makes it the cleanest comparison. On CIFAR-10-C, TEMA's error rises only from 20.20% to 21.05% as batch size falls from 200 to 1. No reported result was found on Lacuna for TEMA on Cityscapes-to-ACDC segmentation, so we run it ourselves.

The strongest method overall is TTN, which uses source labels offline. The file reports TTN raising Cityscapes→BDD-100K mIoU from 43.12% (test-batch normalization) to 47.40%. We treat TTN as an upper reference that has more information than we allow ourselves. We also include RBN-style group-normalization replacement, retrained on the same backbone.

Fairness rules apply to every baseline. All methods share the backbone, the source checkpoint (deep-supervision training applied to every variant, so heads are present even where unused), the test stream order, and the batch size. Every grid or hyperparameter is tuned on one held-out ACDC condition and transferred unchanged.

The key ablation replaces $J$ with final-head entropy and the collapse guard, running the identical search over the identical grid. This isolates whether multi-scale heads matter or whether any confidence score would do. A second ablation shuffles head ordering to break the depth-locality argument.

Beating TEMA means a consistent mIoU gain across ACDC's four conditions and BDD-100K at batch size 1 and 2, at no more than about 2× its compute. Matching TTN within one point without source data would be a strong result.

Why not just…?

Why not just re-estimate statistics from test batches, as AdaBN does? In classification it works well; covariate shift adaptation reduces ResNet-50 ImageNet-C mCE from 76.7% to 62.2%. In segmentation, the Critical Look paper shows statistic updates fall below the unadapted model, and at batch size 1 test-batch normalization reaches about 90% error on CIFAR-10-C (TTN, TEMA). Our method contains this option as $\alpha=1$, so any setting where full re-estimation is best will be visible as the probes selecting it.

Why not just use TEMA's KL-based rectification, which already blends per layer without labels? This is the real threat. KL divergence measures how far statistics have moved, not whether moving them helps predictions. A large style shift in shallow layers may be harmless while a small shift in a deep layer flips classes. The oracle study settles this directly. If TEMA's KL score rank-correlates with true mIoU as well as head agreement does, the probes add nothing and we will report that.

Why not just replace BatchNorm with group normalization? RBN shows strong gains (VisDA-2017 74.0%→82.6%). The same paper notes that removing BN entirely hurts optimization, and that it keeps BN in early layers, so the remaining BN layers still need test-time handling. RBN also requires retraining, while our method applies to existing checkpoints. We include an RBN-retrained backbone with and without our probes to test whether the two are complementary.

Why not just tune input prompts and leave normalization alone? TTAP reaches 66.8% on ACDC. It relies on gradient updates and attention-based backbones, and its reported number is not on our backbones. We run it on the HRNet model for a fair point of comparison. We do not claim to beat gradient-based adaptation; we claim a cheaper, gradient-free repair that could serve as its initialization.

Why not just average the auxiliary heads' predictions at test time? Shallow exits are much weaker (34.3 vs 80.4 mIoU for HRNet exits), so naive averaging should hurt. We include head ensembling as a baseline to confirm the heads' value lies in diagnosing, not voting.

Proposed Experiments

The main benchmarks are Cityscapes→ACDC (fog, night, rain, snow, evaluated separately and as a continual stream) and Cityscapes→BDD-100K, scored by mIoU. CIFAR-10-C at batch sizes 1 to 200 serves as a sanity check against TEMA and TTN's published numbers, using the classification analogue: intermediate classifiers as in Training Deeper Convolutional Networks with Deep Supervision. For the continual stream we report throughput against Hybrid-TTA's reported ~2.0 FPS (Hybrid-TTA) and its 62.2% mean mIoU on Cityscapes-to-ACDC. Because backbones differ, that comparison is contextual.

The decisive check comes first and can stop the project. In the oracle sweep, head-agreement $J$ must rank-correlate with true mIoU better than both TEMA's KL score and final-head entropy, on at least three of the four ACDC conditions. If it does not, the idea is dead regardless of downstream numbers.

The second check is localization. When we artificially corrupt only one stage's statistics on clean Cityscapes, the search should move only that stage's $\alpha$.

The third check is the shallow-versus-deep question. We report oracle and selected $\alpha$ profiles per shift type, and say plainly whether they side with TTN, with RBN, or depend on the shift.

Finally, we report collapse rates, meaning the fraction of streams where predicted class entropy falls below $\tau$, and how often the guard term fires. This shows whether agreement is being bought by predicting road everywhere.