Counting Hypotheses, Not Checkpoints: Similarity-Equalized Priors for Model Merging

When we combine many trained models into one, we usually let every model cast an equal vote. Public model hubs break that assumption. A pool of fine-tuned checkpoints often holds five near-copies of one popular recipe and a single model trained for an unusual skill. Uniform averaging gives the copies five times the say. So does Fisher-weighted merging and the sign vote in TIES-Merging. The rare specialist gets drowned out. Image retrieval faced the same failure fifteen years ago. A photo of a brick wall produces thousands of nearly identical local features, and summing them buries the one feature that identifies the building. The fix there was democratic aggregation: before pooling, reweight the items so that each distinct pattern contributes about equally, however many times it repeats.

This note proposes moving that fix from image features to model averaging. We would compute a weight for each checkpoint by solving the same equalization problem used in Generalized Max Pooling, applied to a similarity matrix between models. We then treat those weights as a Bayesian prior, a prior that is uniform over distinct hypotheses rather than over files, and plug it into Fisher merging and TIES. The claim to test has two parts. Under the right formulation, duplicating a checkpoint $k$ times should leave the merged model essentially unchanged. A specialist's contribution should also survive in pools that are heavily redundant.

Core Claim

The linear form of similarity equalization is exactly clone-invariant. The Sinkhorn form used in most democratic pooling work is not. This distinction is what makes the transfer worth doing rather than obvious.

Let $K$ be an $M\times M$ nonnegative similarity matrix between the $M$ checkpoints in a pool, with entries $K_{mn}$. We use two choices of $K$. The first is agreement of predicted distributions on an unlabeled probe set. The second is cosine similarity of task vectors, meaning each fine-tuned model's weights minus the shared starting weights. Following Generalized Max Pooling, which asks the pooled vector to have the same inner product $c$ with every local descriptor, we solve for model weights $w$:

$$w^\star = \arg\min_{w \ge 0}\; \|K w - c\mathbf{1}\|^2 + \lambda \|w\|^2 .$$

Here $\lambda$ moves smoothly from full equalization (small $\lambda$) to uniform weighting (large $\lambda$). The nonnegativity constraint is our addition. It keeps every weight usable as a prior probability and blocks the negative "spoiler" weights that unconstrained equalization can produce, the effect noted by Breaking the Illusion of Artificial Consensus.

Consider $k$ exact copies of one model that are orthogonal to the rest of the pool. The constraint becomes $k w_{\text{copy}} = c$. The cluster's total weight is therefore $c$, the same weight a singleton receives. Now consider the dampened-Sinkhorn condition $w_m \sum_n w_n K_{mn} = c$ from Second-Order Democratic Aggregation and Triangulation Embedding and Democratic Aggregation. The same cluster ends up with total weight $\sqrt{ck}$, so redundancy still buys influence that grows like $\sqrt{k}$. This calculation predicts a measurable difference between the two families. That prediction is the first thing we test.

The weights $w^\star$ become the model prior $P(m)$. When labeled validation data exists, we multiply the prior by an evidence term, so $\pi_m \propto w^\star_m\, P(D\mid m)$. Without validation data we use the prior alone. The merged parameters are a prior-weighted version of the precision-weighted average in Merging Models with Fisher-Weighted Averaging:

$$\theta = \Big(\sum_m \pi_m F_m\Big)^{-1}\sum_m \pi_m F_m \theta_m .$$

Here $F_m$ is model $m$'s diagonal Fisher information, a per-parameter measure of how much the model's predictions depend on that parameter, and $\theta_m$ are its parameters. For TIES, $\pi_m$ replaces the unit vote each model casts when the consensus sign is elected.

Position Relative to Prior Work

Two papers supply the method and one supplies the target property.

Generalized Max Pooling provides the equalization objective. It shows that equalization weights act as an unsupervised saliency detector, raising CUB-2011 accuracy from 6.0% under sum-pooling to 24.8%. Interferences in Match Kernels explains why summing correlated items corrupts the resulting similarity scores. We reuse both ideas, but over models rather than image patches.

Breaking the Illusion of Artificial Consensus states clone-robustness as an axiom for weighting points in a metric space. Its maximum-entropy construction is intractable, and it is never tested on data. Our nonnegative ridge solve is a cheap continuous candidate for meeting those axioms, which we check empirically.

Two merging papers motivate the gap and serve as the systems we modify.

Fisher merging treats each model as a Gaussian posterior centred on its weights (a Laplace approximation) and weights parameters by curvature. Each model is handled independently, so near-duplicates are counted multiple times.

TIES-Merging identifies redundant updates and sign conflicts as the source of interference. It resolves them by trimming small updates and holding a majority sign vote. That vote is exactly the place where clones dominate. Our proposal leaves both methods' machinery intact and changes only how many votes each checkpoint gets.

The Case for Bayesian Deep Learning frames deep ensembles as an approximation to averaging over a posterior that covers several distinct solutions. If redundant members waste that approximation on one solution, a hypothesis-level prior is the natural correction. We use this framing in a secondary ensemble experiment.

Research Program

We build two components. The first is a solver that, given a pool and a probe set, returns $w^\star$. Its cost is one $M\times M$ nonnegative least-squares problem, which is negligible for pools of tens to hundreds of models. The second is drop-in integrations into Task Arithmetic, Fisher merging and TIES.

The analysis has three parts. First, we prove the clone-invariance of the linear form for exact duplicates. We also bound how much a cluster's weight drifts when its members are near-duplicates with off-diagonal similarity $1-\epsilon$. Second, we compare the linear and Sinkhorn forms empirically. Third, we test which similarity space actually detects redundancy: function-space agreement or task-vector cosine. Task vectors from different tasks are often nearly orthogonal, so they may separate tasks well and still fail to flag redundancy. Function-space agreement on probes is expected to work better, and an ablation tests this.

Proposed Experiments

The main testbed is the eight-task ViT merging suite used by TIES-Merging: Stanford Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397 and SVHN. We add controlled redundancy in three forms.

  1. Exact-recipe clones. We add $k\in\{1,2,4,8\}$ fine-tunes of a single task that differ only in random seed.
  2. Graded overlap. We add fine-tunes on 25–100% overlapping subsets of one task's data, plus fine-tunes on two-task mixtures, so that redundancy is a matter of degree rather than yes-or-no.
  3. Hub-like pool. We assemble a mixture with an uneven, long-tailed number of checkpoints per task.

We replicate the main findings on the GLUE tasks used by Fisher merging and TIES with T5-base, namely RTE, MRPC, SST-2 and others from GLUE, using the same three forms of injected redundancy.

The metrics are as follows.

  • Average accuracy across tasks.
  • Worst-task accuracy, which captures what happens to the drowned-out specialist.
  • Clone sensitivity: the change in accuracy on the non-duplicated tasks as $k$ grows. An ideal method scores zero here.

A secondary experiment builds deep ensembles on CUB-200-2011 with deliberately redundant members. We compare expected calibration error and negative log-likelihood under uniform, evidence-based and democratic-prior averaging. This tests whether the prior also improves predictive uncertainty, not only accuracy.

Baselines

Naive reference. Uniform parameter averaging, plus Task Arithmetic with one global scaling coefficient. These represent "every checkpoint votes once" and will be run on identical pools.

Fisher merging. We compute the Fisher information on the same number of examples per model as the original method. This is the most direct Bayesian competitor. Beating it means higher worst-task accuracy and near-zero clone sensitivity on redundant pools.

TIES-Merging. This is the strongest existing method. Its paper reports average gains of 1.8% and 1.5% over the next-best merging baseline on the ViT vision suite, and 2.3% on T5-base NLP tasks. No absolute accuracy figure for the eight-task vision suite is available on Lacuna. We will therefore re-run TIES ourselves with its published trim fraction, and tune its scaling coefficient on the same validation split and grid we give our method. Beating it means two things. On redundant pools, TIES with democratic votes should exceed plain TIES in worst-task accuracy. On the clean eight-task pool, it should match plain TIES within seed noise.

Ablations. These isolate the key ingredient.

  • Sinkhorn-form weights versus linear-form weights, with everything else fixed. The prediction is $\sqrt{k}$ clone sensitivity for the Sinkhorn form versus roughly zero for the linear form.
  • Task-vector similarity versus function-space similarity.
  • A $\lambda$ sweep, which should smoothly recover the uniform baselines at large $\lambda$.
  • An oracle that knows each checkpoint's true training provenance and gives each distinct recipe one vote.

Why not just…?

Why not just cluster the checkpoints and keep one per cluster? This is the most serious threat. On the exact-clone pools it should match our method. Hard deduplication needs a similarity threshold, and graded overlap has no correct threshold: a model trained on 60% shared data is neither a copy nor independent. The graded-overlap pools are designed to settle this question. We will sweep the threshold and report the best choice made with hindsight. If thresholded deduplication at its best threshold matches us there too, we will report the continuous method as unnecessary.

Why not just learn merging coefficients on labeled validation data, as stacking does? Fitting combination weights to minimize validation loss does down-weight correlated members. When labels for every task are available it may match or beat us, so we include it with the same validation budget. Our method needs only unlabeled probes. The comparison of interest is therefore how close a label-free prior gets to this label-hungry upper reference. A further check is whether the democratic prior improves it when validation sets are small.

Why not just rely on Fisher information, since it already measures importance? Fisher information measures how sensitive a single model is to each parameter. It carries no information about whether another model in the pool already says the same thing. Five seeds of a Cars fine-tune have five similar Fisher matrices, and the sum counts them five times. The clone-sensitivity curve will show this directly.

Why not just run the full ensemble? Running $M$ models multiplies inference cost by $M$, which is the motivation stated in Merging Models with Fisher-Weighted Averaging. Even ensembles over-weight redundant members, which is why the CUB ensemble experiment applies the same prior there.

Decisive Checks

Three results would establish the claim, and the first two can each refute it.

  1. Clone sensitivity. On the exact-clone pools, Fisher merging and TIES should lose accuracy on the non-duplicated tasks roughly in proportion to $k$. Linear-form democratic weights should stay flat. If ours also degrades, the core claim fails.
  2. No cost on clean pools. On the clean eight-task pool, democratic weights should not hurt TIES or Fisher merging.
  3. Graded overlap. On the graded-overlap pools, the continuous method should beat the best hard-deduplication threshold in worst-task accuracy. If it does not, the contribution reduces to the clone-invariance analysis plus a recommendation to deduplicate, and we would report it that way.