Disjoint-Data Weight Averaging: Ensemble Robustness at the Cost of One Training Run

Averaging the weights of several fine-tuned copies of a pretrained network is one of the cheapest ways to get ensemble-like robustness at test time. The result is a single model with no extra inference cost. The catch is training cost. Diverse Weight Averaging, which trains $M$ copies on the full dataset with different hyperparameters and then averages them, names its $M\times$ training bill as its main bottleneck. We propose to remove that bill by changing what makes the copies different. Instead of varying hyperparameters on the same data, each copy is fine-tuned from the shared pretrained starting point on its own slice of the training set, and the slices together cover the data roughly once. The total compute then matches one ordinary training run.

The bet is that different data supplies the "different mistakes" an ensemble needs more cheaply than different hyperparameters do. The shared pretrained starting point keeps the copies close enough in weight space that their average still works. If this holds, weight-averaged ensembles stop being a luxury for well-funded benchmarks and become a drop-in replacement for standard fine-tuning. If it fails, it should fail at a measurable slice size, which would tell us how much data a member needs before averaging breaks.

Core Claim

Let $\theta_0$ be a pretrained network. Let $D_1,\dots,D_M$ be subsets of the training set $D$, each of size roughly $|D|/M$, where consecutive subsets share a fraction $r$ of their examples ($r=0$ means the slices are disjoint). Let $\theta_m$ be the weights obtained by fine-tuning $\theta_0$ on $D_m$ alone. The deployed model is

$$\bar\theta = \frac{1}{M}\sum_{m=1}^{M}\theta_m ,$$

and the total number of gradient steps across all members is held to a budget $B$, measured in units of one standard full-data fine-tuning run.

We claim three things. First, at $B=1$, $\bar\theta$ beats both standard fine-tuning and single-run weight averaging on out-of-distribution accuracy. Second, at moderate $B$, it matches the accuracy of full-data Diverse Weight Averaging at a small fraction of that method's cost. Third, the overlap fraction $r$ controls a real tradeoff with an interior optimum. That tradeoff comes from the bias–variance–covariance–locality decomposition introduced by Diverse Weight Averaging for Out-of-Distribution Generalization (DiWA). Disjoint data lowers the covariance between members' errors, which is good. Smaller slices raise each member's variance and push members further apart in weight space, which raises the "locality" penalty paid when the average leaves the low-loss region.

The first two claims are about the method. The third is a measurable prediction about where it breaks.

What Prior Work Does and Does Not Do

Two papers motivate this proposal and supply most of its method.

DiWA shows that averaging independently fine-tuned members from a shared starting point reduces variance and error covariance. It reports 68.0% average out-of-distribution accuracy on DomainBed against 66.9% for SWAD (single-run checkpoint weight averaging), and finds that accuracy keeps rising as $M$ grows from 20 to 60. Every DiWA member, however, sees the whole training set, so diversity is bought only through hyperparameter noise at full price. We keep DiWA's averaging and its decomposition but move the source of diversity from hyperparameters to data.

Lowering the Pre-training Tax for Gradient-based Subset Training trains learners from an identical starting copy on tiny disjoint random slices, collectively seeing 2–25% of the data. It adds pruning and RandAugment to force diversity, then merges them into a weight-averaged soup. That soup only serves as a warm-up model for scoring examples before a separate subset-selection run, and the slice learners are discarded. Its existence result constrains our design optimistically: learners trained on disjoint slices from a shared starting point can be averaged into something coherent. Our proposal differs in using the merged model as the final model and in studying the full fine-tuning regime rather than a short warm-up.

Three further papers explain why the obvious adjacent ideas are not this one.

Training Data Subset Search With Ensemble Active Learning uses checkpoint ensembles only to pick one subset for one downstream model. It reports a 0.5% top-1 ImageNet gain from an 80% subset. The ensemble there selects data and does not predict.

A Novel Surrogate-assisted Evolutionary Algorithm Applied to Partition-based Ensemble Learning does assign data partitions to ensemble members and finds that searched partitions beat random ones. It works only up to 500 tabular samples with black-box base models and no weight averaging. Its finding that partition structure matters motivates our structured-partition ablation.

Feature Bagging Provides Stability proves stability gains from subsampling feature columns in linear and tree models. We work on the instance axis and in deep networks, which that paper leaves open.

Supporting evidence that data-induced diversity outperforms run-to-run diversity comes from Improving Low Resource Speech Translation with Data Augmentation and Ensemble Strategies. There, ensembles over differently augmented training data gave about 17% BLEU over the average member, while checkpoint ensembles were marginal. That result concerns prediction ensembles, not weight averages, so it motivates our hypothesis without settling it.

Research Program

The work has three parts.

Measuring the tradeoff. We first map the tradeoff the decomposition predicts. Using DomainBed's protocol, we fine-tune members of an ImageNet-pretrained ResNet-50 on partitions of the source-domain training data. We sweep the number of members $M\in\{4,10,20,40\}$ and the overlap $r\in\{0,0.25,0.5\}$, under total budgets $B\in\{1,2,5\}$ and DiWA's own budget. For every configuration we evaluate two models: the averaged model $\bar\theta$ and the prediction-averaged ensemble of the same members. The gap between them directly estimates the locality penalty. Comparing member-to-member error correlation against DiWA members estimates the covariance gain. The output is a map of where disjoint slices stop being averageable as a function of slice size, and we expect this to depend strongly on dataset size. DomainBed's smaller datasets, such as PACS, will reach small slices quickly.

Partition structure. Second, we test whether structure beats random slicing. We compare random partitions, class-stratified partitions, and source-domain-aligned partitions, where each member sees mostly one training domain. The domain-aligned variant is the most interesting: it maximizes data diversity but risks the largest weight-space drift. This tests in deep networks the partition-structure effect seen at small scale in the evolutionary-partition paper.

Stabilizers. Third, we test whether the stabilizers from the pre-training-tax paper (RandAugment on small slices, and greedy selection of members against a validation set) are necessary or just helpful. We prefer uniform averaging as the default because it does not depend on a clean validation set, which that paper flags as a limitation.

Baselines

Standard fine-tuning (ERM). The naive reference is ordinary empirical-risk-minimization fine-tuning on the full source data at $B=1$, run with DomainBed's standard hyperparameter search. No reported result was found on Lacuna for ERM's DomainBed average in this file, so we will report our own run under the identical protocol. Any gain at $B=1$ counts as a win over doing nothing differently.

SWAD. The strongest equal-compute competitor is single-trajectory weight averaging. DiWA reports 66.9% average DomainBed out-of-distribution accuracy for it. It costs one run and is the method a practitioner would actually use instead of ours. We run it at $B=1$ with the same backbone, data splits, and tuning budget. Beating 66.9% at $B=1$ is the headline test.

Full-data DiWA. The strongest existing method overall is DiWA, with a reported 68.0% average out-of-distribution accuracy on DomainBed at $M=20$ or more full runs. We will reproduce it at its published budget and also compare at every smaller budget. Matching 68.0% within run-to-run noise at $B\le5$ would count as beating it on cost.

Prediction ensemble of our members. This reference costs $M\times$ at inference. It separates gains from data diversity (which it captures) from losses due to locality (which only the weight average pays).

Shortened full-data DiWA (key ablation). This is the ablation that isolates our ingredient. It has the same $M$ and the same per-member step count as our method, so it costs the same, but each member draws its steps from the full dataset instead of a private slice. The only difference from our method is whether members' data are disjoint.

Why not just…?

Why not just train one model with SWAD and call it done? SWAD is free and strong. DiWA's comparison shows multi-member diversity adds about one point over it, but only at many times the cost. Whether that point survives at equal cost is exactly the $B=1$ comparison above. If SWAD ties us at $B=1$, our method's case rests only on the intermediate budgets, and we will say so.

Why not just run DiWA with shorter runs on the full data? This is the real threat. Splitting a budget into $M$ short fine-tunes from a shared pretrained start might generate enough diversity on its own, with disjointness adding nothing. The shortened-DiWA ablation settles this directly. If it matches our method, the honest conclusion is "cheap diverse fine-tunes suffice," and the data-partitioning claim is withdrawn. Our hypothesis predicts that disjoint members show lower error correlation at equal steps, and we will measure that directly.

Why not just use the pre-training-tax soup as an initialization and fine-tune one model on the selected subset? That is the pipeline of the pre-training-tax paper itself, where the soup only guides selection. We run it under our budget accounting, counting its warm-up, scoring, and subset training, as an additional comparison. If merged slice learners are good enough to be the final model, the second training phase is wasted compute.

Why not just bag with ordinary prediction averaging? It is likely more accurate than our averaged model, because it pays no locality penalty. But it costs $M\times$ at inference, which deployment settings forbid. We report it as the upper reference, and the gap to it is our measured cost of collapsing to one model.

Decisive Checks

All results are reported as DomainBed average out-of-distribution accuracy with standard errors over three data splits, plus per-dataset accuracy, member error correlation, and the gap between the weight average and the prediction ensemble.

The proposal succeeds if three conditions hold together. The averaged model exceeds SWAD's 66.9% at $B=1$. It reaches DiWA's 68.0% at $B\le5$. And it beats the shortened-DiWA ablation at matched budget.

The locality prediction is falsified if accuracy rises monotonically as overlap goes to zero at every $M$ and shows no interior optimum. It is also falsified if the weight-average-versus-ensemble gap does not grow as slices shrink.

As a secondary, label-noise test, we would fine-tune on CIFAR-100 with 30–50% injected label noise, the noise range studied by Refined Coreset Selection. This checks whether averaging slice members dilutes the memorized noise that each member sees only locally.