Mixtures of Weight Averages: Choosing How Many Models to Keep When Averaging Breaks

Running an ensemble of many fine-tuned networks makes predictions more robust under distribution shift, but every extra member adds a full forward pass at test time. Weight averaging removes that cost: it merges many models into one network and runs it once. It only works when all the models sit in the same low-loss region of parameter space, though. Once some of them drift apart, the average lands somewhere none of them would have chosen. This proposal treats a pool of fine-tuned runs as a mixture of a few groups. Within a group, models can be safely averaged into one network. Across groups, they cannot. The method finds the groups, chooses how many to keep from unlabeled test inputs, averages within each group, and ensembles only the resulting few averaged models. Inference cost is then set by the number of genuinely distinct groups $K$, not by the number of trained models $M$, and $K$ is picked by a rule rather than hand-set.

The practical case is out-of-distribution image classification, where practitioners already train dozens of runs and must choose between paying $M\times$ inference for an output ensemble and accepting the failures of a single average. If the pool really has one group, the method reduces to plain averaging at $1\times$ cost and loses nothing. If it has several, the method should recover most of the full ensemble's accuracy at a small multiple of single-model cost. The main open question is whether the cost-setting number $K$ can be chosen without target labels.

Core Claim

Weight averaging and output ensembling are the two ends of one design space, and the right operating point depends on how many weight-space groups the trained pool contains. Diverse Weight Averaging (DiWA) averages independently fine-tuned runs from a shared pretrained start and explains its gains with a bias, variance, covariance and locality decomposition. In that decomposition, averaging reduces variance but pays a "locality" penalty when the averaged models are far apart; output ensembles do not pay that penalty. DiWA weights every run equally and gives no guidance when locality fails. Topology-informed Ensemble Pruning (TEP) prunes an output ensemble using diversity measured on unlabeled test inputs. The kept members still run separately, and the number kept is fixed by hand at $N/2$, which the authors themselves list as unresolved.

Our claim is that grouping by functional closeness, then averaging within each group and ensembling across groups, beats both at matched inference cost whenever the pool spans more than one region. We further claim that the number of groups can be chosen from unlabeled target inputs by measuring how faithfully the $K$-group system reproduces the full $M$-model ensemble, with a complexity penalty. The selection rule is the new ingredient. The grouping step reuses known techniques.

Research Program

The first step builds a pool and a similarity measure. Following the DiWA protocol and its released code, alexrame/diwa, we fine-tune $M$ between 20 and 60 ResNet-50 runs per DomainBed dataset under mild hyperparameter variation. To create pools that are known to contain several groups, we also add runs from a second pretrained initialization; DiWA's own analysis implies these cannot be averaged with the first set.

Pairwise similarity is measured functionally rather than in raw parameters, borrowing the idea from HC-SMoE, which merges mixture-of-experts sub-networks by comparing their outputs on calibration data rather than their weights or routing scores. Our signal is a mix of two quantities on unlabeled target images. The first is the disagreement between the two models' predictions. The second is the loss increase at the midpoint of a linear interpolation between their weights. We cannot compute that loss without labels, so we approximate it by the entropy and prediction flips of the midpoint model relative to its endpoints. Hierarchical clustering with average linkage then gives a nested family of groupings for every $K$ from 1 to $M$.

The second step chooses $K$. For each $K$ we average the weights within each group, giving $K$ "soups," and ensemble their outputs with weights proportional to group size. Let $p_M(x)$ be the full $M$-model output ensemble's predicted class distribution on an unlabeled target input $x$, and let $q_K(x)$ be the $K$-soup system's. We score

$$ S(K) = \frac{1}{n}\sum_{i=1}^{n} \mathrm{KL}\big(p_M(x_i)\,\|\,q_K(x_i)\big) + \beta\,\frac{K \log n}{n}, $$

where $n$ is the number of unlabeled target inputs and $\beta$ is a single constant set once on a held-out DomainBed dataset and then frozen. We pick the $K$ that minimizes $S(K)$. The penalty term has the form of the Bayesian information criterion: each extra group behaves like one more component whose benefit must exceed a $\log n / n$ cost. In spirit this follows the message-length view of mixture selection used in adaptive color mixtures for segmentation and the Gaussian-mixture description-length bounds, applied here to models rather than data points.

The rule has a known weakness. It measures fidelity to the full ensemble, not accuracy, so it can only be as good as that ensemble. We accept this deliberately, because the full ensemble is the expensive target we are trying to imitate cheaply.

The third step is analysis. On every dataset we report the selected $K$ next to the accuracy-versus-$K$ curve computed afterward with target labels. We also test a prediction from the DiWA decomposition: the gain over a single average should grow with the measured spread within the pool, and the selected $K$ should be 1 whenever the pool comes from a single initialization and is tightly clustered.

Baselines

The naive reference point is a single fine-tuned model (empirical risk minimization) at $1\times$ cost, run with the same DomainBed protocol and hyperparameter ranges as every other method.

The strongest zero-overhead competitor is DiWA's uniform average of all $M$ runs, which reports 68.0% average DomainBed accuracy versus 66.9% for SWAD (DiWA). It is the right opponent because it already turns a pool into a single network and is the method our approach reduces to when $K=1$. We rerun it on exactly our pools, using the same trained runs, rather than quoting the number. On single-initialization pools, beating it means higher average accuracy at a mean $K$ of 2 or 3. On mixed-initialization pools, a clear margin is expected.

The strongest overhead-reducing ensemble competitor is TEP, which reports 52.1% on TerraIncognita against 51.3% for a full uniform ensemble (TEP). We run TEP on the same pool with its retained count set to our selected $K$, so inference cost is identical. We also run it at its default $N/2$. Beating it means matching or exceeding its accuracy at the same $K$, since our $K$ members are averages of groups rather than individual runs.

The cost-unbounded reference is the full $M$-model output ensemble. Recovering at least 90% of its gain over a single model at $K \le 3$ would count as success.

Three ablations isolate the key ingredients. The first replaces our selection rule with $K$ chosen on in-domain validation accuracy. The second replaces functional grouping with random groups of the same sizes. The third replaces it with k-means on raw flattened weights. All variants use the same pool and the same number of unlabeled target images, and they receive no separate tuning budget, since $\beta$ is fixed once.

Why not just…?

Why not just pick $K$ by sweeping it on validation data? Averaging is cheap, so a sweep is cheap, and this is the most serious alternative. The concern is that DomainBed validation data comes from the training domains, and the open-domain results in Discover, Hallucinate, and Adapt show that choices tuned on seen domains can favor easy modes. Our first ablation answers this directly. If in-domain validation selects the same or a better $K$ than our rule on most datasets, the unlabeled-target rule adds nothing, and we will report the method as "grouped soups with validation-chosen $K$."

Why not just run TEP and tune its retained count? TEP keeps individual models, so each retained member costs a full forward pass. It also computes diagonal Fisher information on test data, which is expensive for large models. At equal $K$, a member that averages a whole group should carry more variance reduction than a single run. The matched-$K$ comparison above tests this. If TEP wins at equal cost, the averaging step is not pulling its weight.

Why not just distill the full ensemble into one network? Distillation also reaches $1\times$ inference and is a real threat. Its costs are an extra training run, and the risk that the single student fails to absorb diversity across groups, which is the same gap the neural-thickets study found when searching for one strong expert against a 50-model ensemble. We found no reported distillation result for DomainBed on Lacuna, so we run it ourselves: one ResNet-50 student trained on the full ensemble's soft labels over the training domains, using the same total training compute as our method. We compare it at $1\times$ against our method at its selected $K$, and against our method forced to $K=1$.

Proposed Experiments

The main experiment covers all DomainBed datasets (PACS, VLCS, OfficeHome, TerraIncognita, DomainNet) under the leave-one-domain-out protocol. Each dataset is tested with two pools: single-initialization and mixed-initialization. We report accuracy, inference cost as $K$, and the gap to the labeled-oracle $K$.

The first decisive check is the mixed-initialization pool. There, uniform DiWA should collapse, and the method should select $K \ge 2$ and recover near-ensemble accuracy. If it selects $K=1$, the similarity measure is broken.

The second check is the single-initialization pool. There, the method should rarely beat DiWA by much, and should choose $K=1$ when the pool is tight. Choosing large $K$ without any gain would show the penalty $\beta$ is miscalibrated.

The third check repeats the selection-rule ablation across datasets. It decides whether the unlabeled-target criterion is a contribution or merely a convenience. If time permits, we would transfer the same rule to the 8-task CLIP merging setting, where WEMoE reports 89.4% with ViT-B/32 while keeping every task vector as a separate expert. That would test whether grouping task vectors can shrink the expert count without losing accuracy.