Hybrid Normalization as a Mergeability Device for Multi-Domain Segmentation

When several copies of one pretrained network are each adapted to a different target domain, such as daytime Indian roads, foggy European streets, or night driving, their weights can often be averaged into a single model that handles all domains at the cost of one. This is weight-space ensembling. The averaged model usually pays an accuracy "tax" relative to each specialist. A large share of that tax lives in the normalization layers and in the shallow layers, where specialists drift apart to absorb each domain's lighting, contrast, and color.

Hybrid normalization addresses exactly that kind of variation. These layers split or blend two operations. Batch normalization standardizes each channel using statistics pooled over many images. Instance normalization standardizes each image on its own, which strips its style. This note proposes building domain specialists on a hybrid-normalized backbone so that they disagree less where merging hurts most. It also derives the merging rule that hybrid layers need: their per-channel BN-versus-IN gates are nearly binary, and plain averaging breaks them.

Core Claim

The claim has two parts, and each can fail on its own.

The first part is about the architecture. Instance-normalized channels compute their statistics on the fly and hold no stored running statistics, and they remove per-image style before it reaches later layers. Because of this, specialists adapted from a hybrid backbone should show smaller shallow-layer weight changes and smaller layer-by-layer activation mismatch after merging. They should therefore pay a smaller merging tax than specialists adapted from an otherwise identical batch-normalized backbone.

The second part is about the merging rule. In learnable hybrids, each channel has a gate $\rho \in [0,1]$ that mixes the two outputs as $\rho\,\mathrm{BN}(x) + (1-\rho)\,\mathrm{IN}(x)$. Batch-Instance Normalization reports that these gates settle near 0 or 1, rarely in between. Averaging a gate that one specialist pushed to IN with a gate another specialist pushed to BN lands near 0.5, a state gradient descent almost never chooses. We propose an "invariance-union" rule instead. The reasoning is that a channel that any specialist chose to style-normalize carries appearance that varies across domains, and the merged model must face all domains at once. Each channel's gate is therefore set to the specialists' minimum:

$$\rho_c^{\text{merged}} = \min_k \rho_c^{(k)},$$

where $c$ indexes channels and $k$ indexes specialists. The affine scale and shift of channel $c$ are then averaged only over specialists whose gate falls on the same side, so that scale parameters learned for BN-normalized inputs are not applied to IN-normalized ones.

Relation to Prior Work

The closest existing work is Training-Free Model Merging for Multi-target Domain Adaptation. It averages the weights of single-target segmentation specialists and derives a Gaussian-prior estimator for merging BN running means and variances, since naive averaging of those statistics misrepresents the mixture. It supplies our merging pipeline and our benchmark. It treats normalization only as a container of statistics on plain-BN backbones, whereas we change the normalization itself and handle gates and buffer-free channels.

Two diagnostic papers motivate the proposal. Vanishing Feature shows that merged networks lose input-dependent signal across depth while input-independent bias accumulates. It fixes this after the fact by keeping early layers unmerged or by recalibrating variances. LOT Merging names layer-wise "feature drift" as the cause of the tax and solves for merged weights and LayerNorm/BN scale factors in closed form from 16–64 exemplars. Both patch the merged model; neither asks whether per-instance normalization inside the architecture prevents the drift to begin with.

LiNeS both motivates and threatens the proposal. It finds that shallow task-vector updates are destructive and scales the updates down linearly toward the input. That is the same shallow-style/deep-semantics split that IBN-Net used to place IN only in early blocks. If LiNeS on a BN backbone matches our results, the architectural route adds nothing.

The hybrid-normalization papers supply the mechanism. IBN-Net uses a static 50/50 IN/BN channel split in early blocks. BIN learns per-channel gates. MetaBIN meta-learns gates so they do not overfit source styles. None of the three considers merging. Fisher merging and RegMean give weighting rules for dense weights, and we reuse Fisher weighting as a gate-averaging competitor.

Research Program

We build three backbones that differ only in normalization: plain BN ResNet-101, static IBN ResNet-101 (IN in the first three residual groups), and BIN ResNet-101 with gates. Each is ImageNet-pretrained under the same schedule. From each backbone, we adapt one specialist per target domain using the single-target adaptation recipe of the multi-target merging paper. We then merge the specialists with its parameter-plus-buffer procedure. For the BIN backbone we additionally apply the gate rule.

For every merged model we measure three things. The first is per-target mIoU (mean intersection over union) and its harmonic mean across targets. The second is the merging tax, defined as each specialist's mIoU on its own target minus the merged model's mIoU there. The third is two mechanistic quantities: the norm and pairwise cosine similarity of specialists' weight changes per layer, and LOT-style feature drift per layer, measured as the activation distance between merged and specialist on that specialist's domain. The mechanistic prediction is that hybrid backbones reduce drift specifically in the IN-bearing early groups, and that the tax reduction tracks this drift reduction. For BIN we also record the gate flip rate, meaning how often specialists land on opposite sides of 0.5. If flips are rare, the gate rule is moot, and we will report that.

The benchmark follows the multi-target paper. Sources are GTA5 and SYNTHIA; targets are Cityscapes, IDD, ACDC, and DarkZurich. These targets span geography, weather, and illumination, which is the kind of variation IN is designed to remove. A cheaper classification replication uses Office-Home's four style domains, where BIN already outperforms BN in mixed-domain evaluation.

Baselines

The naive reference is uniform averaging of all parameters and buffers on the BN backbone. RegMean reports this losing 55% relative performance on RoBERTa emotion classification versus 12% for RegMean, so it should be clearly beaten.

The strongest existing method is the parameter-plus-buffer merging of the multi-target paper on a plain BN ResNet-101. That paper reports 56.3% mIoU for the merged model, against 56.2% for pooled "Data Combination" training, and a +5.6% gain in harmonic-mean mIoU over single-domain models. We rerun it ourselves on the same pretraining, adaptation recipe, and targets, rather than quoting those numbers across setups. Beating it means a higher harmonic-mean mIoU and a lower worst-target tax, with differences larger than the seed-to-seed spread over three adaptation seeds.

The second strong competitor is LiNeS depth scaling applied to the same BN specialists. Its two coefficients are tuned on a held-out validation split with the same search budget we give the gate rule. We also run the two together, IBN plus LiNeS, to test complementarity. LOT Merging on BN specialists, given the same 64 exemplars per domain, is a third competitor targeting feature drift directly.

Two ablations isolate the key ingredient. The first gives IBN the same fraction of IN channels but places them in randomly chosen layers across depth. This separates "fewer stored statistics" from "style removed early." The second runs the BIN backbone with gates merged by plain averaging and by Fisher-weighted averaging instead of the invariance-union rule.

Pooled multi-target training and output ensembling of the specialists are reported as upper references with their compute and latency costs stated.

Why not just…?

Why not just average the gates like any other parameter? If specialists rarely flip gates, averaging is fine and our rule is unnecessary. The gate flip rate and the average-versus-union comparison settle this directly. BIN's bimodality result predicts that where flips do occur, averaging produces mid-range gates the network never trained with. That prediction is untested for merged models.

Why not just use LiNeS on an ordinary BN backbone? This is the most serious threat, because it exploits the same shallow-layer insight with no architectural change and no retraining of the backbone. If BN plus LiNeS matches IBN merging on harmonic-mean mIoU and worst-target tax, we will conclude that the effect is purely parameter-space. A hybrid backbone would then be justified only by its own generalization gains, such as IBN-Net's 22.2% to 29.6% GTA5→Cityscapes mIoU without fine-tuning. The combined IBN-plus-LiNeS run tells us whether the two compound.

Why not just use static IBN-Net and skip learnable gates entirely? Static IBN has no gates to disagree about, so it sidesteps the gate problem. If static IBN merges as well as BIN under the union rule, static IBN is the better recommendation. The proposal is designed so that this outcome still answers the primary question about whether hybrid normalization reduces merging tax.

Why not just train one domain-generalized hybrid model on the source and skip adaptation and merging? The source-only IBN model is evaluated on all four targets. If it rivals the merged specialists, adaptation adds little. We expect it to fall short on severe shifts like DarkZurich, but no reported result found on Lacuna for IBN-Net on IDD, ACDC, or DarkZurich.

Decisive Experiments

The central check compares static IBN merged with the existing parameter-and-buffer procedure against BN merged the same way and BN plus LiNeS. It uses three seeds on the four-target segmentation benchmark and compares the tax on each target. The hypothesis is supported if IBN reduces the tax beyond seed variance, and if the reduction is concentrated where per-layer drift falls in the early groups. The random-placement ablation must lose most of that gain for the "style removed early" explanation to hold.

The secondary check compares the union rule against averaging and Fisher weighting on BIN specialists. It is reported together with the gate flip rate, so that a null result can be attributed to rare flips rather than a bad rule. The Office-Home replication tests whether either finding depends on segmentation.

A clean negative is also informative. That outcome would be IBN no better than BN plus LiNeS, with gates that rarely flip, and it would show that normalization choice is irrelevant to mergeability once depth scaling is applied.