Do Normalization Fixes Survive Fair Tuning?

Normalization layers rescale a network's internal activations so training is stable. They also throw information away. Batch normalization (subtract the batch mean, divide by the batch spread) and instance normalization (the same, but per image) remove statistics that often carry real signal, such as texture, contrast and identity cues. A line of work in cross-domain person re-identification, the task of matching the same pedestrian across cameras never seen in training, builds layers that restore the lost information. These methods report large gains. Each gain, though, is measured with its own training recipe, its own hand-set constants and a baseline that nobody re-tuned. We propose to measure how much of each gain survives when every method, including plain batch normalization, gets the same tuning budget and is selected without looking at the test domain.

Doing this honestly is expensive, and that cost is what has kept it undone. The proposal therefore has a second question. Can a cheap single-run hyperparameter tuner, which adjusts hyperparameters by gradient during one training run, stand in for costly repeated search? There is a specific reason to doubt it here. The best-known cheap tuner justifies its core approximation by the whitening effect of batch normalization. The layers under study remove or dilute exactly that effect. If the tuner works worse on instance-normalized networks, a "fair" cheap protocol would quietly handicap the very methods it is meant to judge.

Core Claim

We claim two things, both testable. First, under a matched tuning budget and non-oracle model selection, the advantage of hybrid and restitution normalization over a properly tuned batch-norm baseline will shrink substantially relative to published margins, and the method ranking will change. Second, a single-run hypergradient tuner is a trustworthy substitute for outer-loop search only if its tuning error is uncorrelated with how much instance normalization a network contains. We expect that correlation to be positive, and we will measure it directly.

What is new is not a normalization layer. It is a budget-matched, leak-free comparison of these layers, together with an audit of whether the cheap tuner used to make such comparisons affordable is itself biased by normalization type.

Research Program

Methods compared. The subjects are the methods that motivate the study.

  • BIN (Batch-Instance Normalization) mixes batch and instance normalization per channel through a learned gate. It needs a hand-picked 10x learning-rate multiplier on that gate, which its authors say must be re-tuned for new tasks.
  • IBN-Net (Two at Once: IBN-Net) fixes a 50/50 channel split in shallow layers.
  • SNR (Style Normalization and Restitution) recovers identity signal from the residual that instance normalization discards. Its auxiliary loss weights are set by trial and error.
  • MetaBIN (Meta Batch-Instance Normalization) meta-learns the BIN gate across held-out source domains.
  • ReNorm (Rethinking Normalization Layers for DG ReID) freezes the per-channel scale and shift (the affine parameters) at 1 and 0 and simulates unseen-domain statistics.

All of these methods were reported with supervised ImageNet ResNet-50 backbones and method-specific recipes. This study will not modify any of them.

Protocol. The protocol borrows its structure from three existing studies.

  • Are GANs Created Equal? showed that equal random-search budgets erased most reported differences among GAN variants. From it we take the reporting format: expected best validation score as a function of the number of trials.
  • Rethinking the Evaluation Protocol of Domain Generalization showed two leaks in domain-generalization benchmarks: supervised ImageNet pretraining and target-domain ("oracle") model selection. Plugging them reshuffled method rankings. From it we take its two fixes. Every method is also trained from a self-supervised initialization, and selection uses only a held-out source domain.
  • Personalized Benchmarking with Ludwig warns that forcing one fixed recipe on every architecture penalizes those with different optimization needs. This is why we tune per method rather than fix a single shared recipe.

Search space. The space is small, continuous and shared: base learning rate, general weight decay, and a separate weight decay on normalization scale and shift parameters. Where a method has them, it also includes the gate learning-rate multiplier and the auxiliary loss weight. The separate affine decay also lets us test ReNorm's hard freeze against a tuned soft penalty. Every method gets the same search space size and the same number of trials.

Cheap tuner. The single-run tuner follows Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters. It splits data into one part for weight updates and another for hyperparameter updates. It takes greedy one-step validation gradients and replaces the curvature matrix with the identity, an approximation its authors credit to batch normalization. They report about 30% overhead over one run. For learning rates we add hypergradient descent, which adapts the step size from the dot product of successive gradients. The tuner's validation split is the same held-out source domain used for selection, so it sees no target data.

Tuner audit. For each method $m$ we define tuning regret as

$$R_m = S_m^{\text{search}} - S_m^{\text{online}},$$

where $S_m^{\text{search}}$ is the held-out target mean average precision (mAP) of the best configuration found by full outer-loop search, and $S_m^{\text{online}}$ is the mAP from the single-run tuner.

Alongside regret we measure a direct diagnostic, the alignment between two estimates of how a small hyperparameter change affects validation loss:

  • the tuner's greedy hypergradient;
  • a finite-difference estimate, obtained by perturbing the hyperparameter and training a few hundred steps.

We also measure the top curvature eigenvalue by power iteration. Our prediction is that alignment falls and regret rises with the fraction of channels routed through instance normalization. That fraction is known exactly for IBN-Net and read off the learned gates for BIN and MetaBIN.

If the prediction holds, we test a minimal repair. We replace the greedy one-step estimate with a short multi-step lookahead, only for the normalization-related hyperparameters, and check whether regret evens out.

Benchmarks. We use multi-source leave-one-out person re-identification. Market-1501, MSMT17 and CUHK03 serve in turn as sources or targets, and PRID, GRID and VIPeR serve as small unseen targets, matching the suites used by ReNorm, SNR and MetaBIN. Office-Home provides a classification check outside re-identification, since BIN reports gains there. The visible-infrared methods MRCN and IDKL are out of scope. They use two-stream architectures whose normalization cannot be swapped cleanly, and adding them would blur the comparison.

Baselines

Do-nothing reference. This is a ResNet-50 with ordinary batch normalization trained on each method's published recipe without re-tuning. It reproduces the comparison the literature actually made. SNR's reported Market-1501 to DukeMTMC baseline of 19.8% mAP, against SNR's 33.6%, is the kind of gap this row lets us re-examine (SNR).

Tuned batch-norm baseline. The central competitor is the same batch-norm ResNet-50 given the full matched budget of outer-loop random search over the shared space. If restitution gains largely came from undertuned baselines, this row is where they disappear.

Strongest normalization methods. These are MetaBIN and ReNorm, each run under both its published recipe and the matched-budget protocol. The reference numbers from the file are:

  • MetaBIN reports an average domain-generalization mAP of 73.6%, against 66.4% for SNR and 61.8% for DualNorm (MetaBIN).
  • ReNorm reports 25.6% mAP on MSMT17 and 81.3% on PRID under leave-one-out evaluation (ReNorm).

We first reproduce these within about one point under their own recipes before applying the protocol.

Strongest cheap-tuning competitor. For the tuner question, the competitor is multi-fidelity search. BOHB and Hyperband prune weak configurations on short runs, and BOHB adds a Bayesian model. BOHB gets the same total GPU-hours as the single-run tuner, roughly 1.3 full runs, and also three times that. No reported result was found on Lacuna for BOHB or Hyperband on person re-identification. The only head-to-head tuning numbers in the file are from Self-Tuning Networks: on CIFAR-10, a validation loss of 0.575 against 0.636 for Bayesian optimization, and on Penn Treebank, a perplexity of 70.30 against 72.13. Both are outside this domain.

Ablations. Two ablations isolate the key ingredients. The first is ReNorm with frozen affine parameters versus ReNorm with a tuned affine weight decay. This tests whether the hard freeze is needed or is just an untuned regularizer. The second applies the multi-step lookahead fix to all hyperparameters versus normalization hyperparameters only. This tests whether the bias is specific to normalization.

What counts as a result. Normalization methods "beat" the tuned baseline only if they win at matched budget on the held-out targets, averaged over three seeds. A cheap tuner "beats" multi-fidelity search if, at equal compute, its method ranking agrees better with the full-search ranking.

Why not just…?

Why not just give every method the same fixed recipe and skip tuning? This is the cheapest uniform protocol, and Ludwig explains why it is unfair. BIN's own authors needed a 10x gate learning rate that a shared recipe would not supply (BIN). We still include the fixed-recipe ranking. If it matches the tuned ranking (Kendall rank correlation of at least 0.8), the objection wins and we will say that tuning was unnecessary for this family.

Why not just run equal-budget random search or BOHB for everyone, as the GAN study did? This is the real threat. At re-identification scale it may be affordable, and full random search is our own gold standard. Two things argue against making it the protocol. The full grid of methods, targets, two initializations and seeds multiplies the cost into thousands of runs. And no evidence in the file shows that multi-fidelity pruning keeps rankings intact when methods differ mainly in late-training generalization. We therefore run full search on a subset, two leave-one-out splits with one initialization and about 24 trials per method, and use it to score both cheap tuners. If low-budget BOHB matches full-search rankings as well as the online tuner does, we will recommend BOHB and report the online tuner as unnecessary.

Why not just use a single-run hypergradient tuner and trust it, since it matched grid search before? That evidence comes from MNIST, SVHN and CIFAR-10 networks with batch normalization (Luketina et al.). Those authors tie their identity-curvature shortcut to batch-norm whitening, and the methods here partly remove it. Trusting the tuner blindly could build a new bias into the benchmark. The regret-versus-instance-fraction correlation is the experiment that settles this.

Why not just adopt MetaBIN's meta-learning to tune the gates? MetaBIN tunes only the gate, needs labeled multi-source splits, and leaves learning rate, decay and loss weights at hand-set values. It is one of the methods being evaluated, not a way to evaluate the others. Its two-level meta-learning loop also costs more than one greedy hypergradient step per ten updates.

Decisive Experiments

Gain survival. For each normalization method, we compute the ratio of its matched-budget advantage over tuned batch norm to its advantage under published recipes. This is done separately for supervised and self-supervised initializations and plotted against trial budget, from 1 to 24 trials. We call the literature's attribution substantially confounded if the median ratio falls below 0.5. We call it sound if the ratio stays above 0.8.

Ranking shift. We compare the method ranking under the published recipe with the ranking under matched tuning, using Kendall's tau. Following Rethinking DG Evaluation, we separately report how much of any shift comes from removing supervised pretraining and how much from equalizing tuning.

Tuner bias. Across the five methods and plain batch norm, we test whether single-run regret $R_m$ and hypergradient alignment correlate with the instance-normalization fraction. A significant positive correlation of regret with that fraction, together with lower alignment for instance-heavy models, confirms the concern. The prediction fails if regret is flat across methods.

Repair. If bias appears, we test whether the normalization-only multi-step lookahead brings every method's regret within one mAP point of full search, at under twice the cost of the greedy tuner.

Affine question. We check whether a tuned affine weight decay matches or beats ReNorm's hard freeze on MSMT17 and PRID. If it does, the freeze was a crude stand-in for a regularizer that could have been tuned.

Transfer check. The same gain-survival and tuner-bias analyses are repeated on Office-Home, to see whether the conclusions extend beyond person matching.