Averaging the weights of several trained networks is a cheap and reliable way to get a better model. The average might be taken over checkpoints from one run, or over separate runs that start from the same pretrained initialization. Networks trained only on simulated or rendered images seem like an ideal use case, because it costs almost nothing to train many of them with different rendering randomizations. There is a catch. Most vision backbones use batch normalization, a layer that rescales each feature channel using a running mean and variance recorded during training. Those stored statistics belong to the individual networks, not to their average. They were also computed on synthetic images, not on the real images the model will face. So an averaged simulator model enters the real world with normalization that is wrong in two ways at once.
This note proposes to measure how those two errors combine, and to test a gradient-free fix. The fix freezes the averaged weights and recomputes every normalization statistic from a few hundred unlabeled real images. Our central hypothesis is specific. When networks trained on differently randomized simulators are averaged, their internal features stay well aligned on synthetic inputs but fall out of alignment on real inputs. The averaged model's activations therefore shrink layer by layer much more on real data than on synthetic data. If that holds, a real-data normalization pass after averaging should give gains that are larger than the sum of its parts. Averaging diverse simulator runs would help on real images only after this pass, and the pass would help an averaged model more than it helps a single model.
Core Claim
Two lines of work each handle half of the problem, and no one has looked at the combination. On the averaging side, Averaging Weights Leads to Wider Optima and Better Generalization and Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging established that averaged weights need a fresh forward pass to recompute normalization statistics. Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits showed that stale statistics are the main bottleneck for long-window weight averaging. REPAIR identified the underlying mechanism. When two networks' neurons are imperfectly correlated, interpolating their weights shrinks activation variance layer by layer. Renormalizing the activations recovers most of the lost accuracy. All of these papers recompute statistics on the training distribution.
On the domain-shift side, AdaBN recomputes normalization statistics on unlabeled target images, with no gradient steps. It does this for a single trained network and never averages weights. SWAD averages weights densely within one run to improve out-of-distribution generalization, but recalibrates on source data because it assumes the target domain is never seen.
Our claim is that the gap between simulated and real images changes the averaging problem itself, not just the choice of calibration data. REPAIR's variance collapse depends on how correlated the endpoint networks' neurons are. That correlation is a property of the input distribution. We expect it to be high on synthetic images, which all endpoints were trained on. We expect it to be substantially lower on real images, where each endpoint has picked up different simulator shortcuts. If this is right, three things follow. A model averaged from diverse simulator runs will look healthy on a synthetic validation set and fail on real data. Recalibrating on synthetic data, as SWA and SWAD do, cannot repair that failure. And a small unlabeled real-image pass will recover much of it. The new object is a "real-recalibrated simulator soup." A soup here means a weight average of several networks fine-tuned from the same pretrained initialization, each under a different domain randomization. The new measurement is how variance collapse depends on the input domain.
Research Program
The first stage is diagnostic. We fine-tune K ResNet-50 models, with K up to eight, from one ImageNet initialization on VisDA-C synthetic renders (VisDA). Each model sees different rendering-style augmentation: color and texture randomization, blur and noise, and crop and viewpoint jitter. We average their weights. For every layer we then record two quantities, separately on held-out synthetic images and on unlabeled real COCO images. The first is the ratio of the averaged model's activation variance to the mean of the endpoints' variances. The second is the channel-wise correlation between endpoints. Both are measured before any statistics are reset. The prediction is falsifiable: the real-data variance ratio should fall with depth much faster than the synthetic one, and the gap should grow as the randomizations diverge.
The second stage is the method, and it is deliberately minimal. We freeze the averaged weights and run N unlabeled real images through the network in training mode, so that every batch-normalization layer recomputes its mean and variance. We then evaluate. In batch-normalized networks this reset is the natural analogue of REPAIR's renormalization. What distinguishes it from REPAIR is the choice of statistics to match. REPAIR matches an interpolation of the endpoints' in-distribution statistics. We match statistics of the target domain itself.
We will compare three calibration sources: synthetic, real, and a 50/50 mix of the two. Together they settle the open question the file raises about which distribution should drive the post-averaging pass. We will sweep N from 16 to 5,000 images. The small end is motivated by the finding in Toward Data Efficient Model Merging between Different Datasets without Performance Degradation that about ten samples per class suffice for recalibration across datasets. It is also motivated by the finding in Adversarial Training with Stochastic Weight Average that inexpensive calibration data suffices after averaging.
The third stage tests generality and one downstream use. We repeat the study on GTA5→Cityscapes segmentation (VisDA-S) with a batch-normalized DeepLab-style network. We also run a LayerNorm backbone, a ViT or ConvNeXt, as a control. REPAIR reports that LayerNorm networks are immune to variance collapse, so on these models the soup should work without any calibration pass, and the super-additive effect should disappear. Finally, following Gap 3 in the file, we use the recalibrated soup as the frozen teacher in a mean-teacher self-training run in the style of Self-ensembling for visual domain adaptation. The question is whether it produces more accurate pseudo-labels than a teacher that tracks its normalization statistics online.
Baselines
The naive reference is a single ResNet-50 trained on synthetic data and evaluated on real images with its synthetic statistics. The VisDA paper reports that an unadapted AlexNet falls from 99% on synthetic data to 28% on real data. We will re-measure this reference for our own backbone rather than rely on the AlexNet figure.
The first strong competitor is AdaBN applied to the best single endpoint and to each endpoint in turn. It is the strongest zero-gradient method in the file and isolates the value of the domain correction alone. On Office-31, AdaBN reports 76.7% average accuracy, against 75.5% for its baseline. We found no reported result on Lacuna for AdaBN on VisDA-C. We will run it with exactly the same real calibration images, the same N, and the same batch size as our method.
The second competitor is SWAD within one run, with its default synthetic recalibration. This isolates the value of flatness alone. SWAD reports +3.6 points over ERM on DomainBed, but we found no reported VisDA-C number on Lacuna. The third is the same soup recalibrated on synthetic data, which is exactly what current averaging practice would do. Every run shares the same number of endpoints, training steps, and augmentation budget, and hyperparameters are selected on a small held-out labeled real set that all methods share.
The key ablations form a two-by-two design: single model versus soup, crossed with synthetic versus real calibration. Adding SWAD with real calibration separates the diversity that comes from different randomizations from the flatness that comes from within-run averaging.
The strongest system-level reference is the output ensemble of the K endpoints, each recalibrated with AdaBN. It costs K times the inference compute. Full unsupervised domain adaptation gives the ceiling. On VisDA-C, Domain-Specific Batch Normalization reports 80.2% average accuracy, against 65.0% for its MSTN baseline, and self-ensembling reports 92.8%. We do not expect a gradient-free pass to reach these numbers.
Beating the baselines means three things. First, the soup with real calibration exceeds the best single endpoint with AdaBN by clearly more than the seed-to-seed spread. Second, the improvement from real calibration is larger for the soup than for single models, which is the super-additivity claim. Third, the soup closes most of the gap to the K-model ensemble at one model's inference cost.
Why not just…?
Why not just run AdaBN on one simulator-trained model? It is cheaper and needs no averaging, and its Office-31 gain is real but small (+1.2 points). This proposal is a direct test of it. If the soup with real calibration does no better than the best endpoint with AdaBN, the averaging half of the idea contributes nothing and we will report that. Our hypothesis predicts a gap that widens with the diversity of the randomizations, because a single model cannot capture what several differently randomized models learn.
Why not just ensemble the model outputs? This is the most serious threat. Output ensembles do not suffer variance collapse and usually beat weight averages. We concede that the soup's value is one-model inference cost, which matters for embedded perception, the setting the simulator-trained detectors in Driving in the Matrix target. The comparison at matched endpoints will show exactly how much of the ensemble's gain the calibrated soup keeps. If it keeps little, the method has no practical case.
Why not just switch to LayerNorm or a transformer backbone? REPAIR's LayerNorm result suggests this avoids variance collapse. It does not remove the synthetic-to-real shift, and much deployed robotics and driving perception still uses batch-normalized convolutional networks. We include a LayerNorm backbone as a control, not a competitor. If an uncalibrated LayerNorm soup matches our batch-normalized soup with real calibration, that is useful guidance for practitioners, but it also narrows our claim to batch-normalized architectures.
Why not just run full domain adaptation or translate pixels with a GAN, as PixelDA does? These methods reach higher accuracy but need adversarial or self-training loops, joint access to source and target data, and many gradient steps. Our method needs a few hundred unlabeled images and no gradients. The self-training experiment tests whether the two approaches are complementary: the recalibrated soup should serve as a better starting point, not a replacement.
Decisive Checks
Three results would falsify the idea. The first is that the per-layer variance ratio of the uncalibrated soup is no worse on real images than on synthetic ones, meaning the collapse does not depend on the domain. The second is that the gain from real calibration is the same for single models and soups, meaning no super-additivity. The third is that synthetic calibration performs as well as real calibration on the soup. If instead real calibration with N≈100 unlabeled images recovers most of the ensemble's accuracy on VisDA-C and GTA5→Cityscapes, and this effect vanishes on the LayerNorm control, then domain-dependent variance collapse is the mechanism. That would give practitioners a concrete rule: recalibrate averaged simulator models on the deployment domain, never on the simulator.
References
- Averaging Weights Leads to Wider Optima and Better Generalization
- Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging
- Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
- REPAIR: REnormalizing Permuted Activations for Interpolation Repair
- Revisiting Batch Normalization For Practical Domain Adaptation
- SWAD: Domain Generalization by Seeking Flat Minima
- VisDA: The Visual Domain Adaptation Challenge
- Toward Data Efficient Model Merging between Different Datasets without Performance Degradation
- Adversarial Training With Stochastic Weight Average
- Self-ensembling for visual domain adaptation
- Domain-Specific Batch Normalization for Unsupervised Domain Adaptation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks