When several fine-tuned copies of one network are combined into a single set of weights ("model merging"), someone has to decide the order. Should all models be averaged at once? Should they be merged in pairs and then pairs of pairs? Which model should every difference be measured from? Current practice answers these questions by convention. Recent evidence shows the convention matters. Meta-Merging by Checkpoint Nowcasting reports that a learned merger that works well on pairs falls behind plain arithmetic averaging once it is forced into a hierarchical, tree-shaped schedule for 4 or 8 models. This note proposes to read the merge schedule off the geometry of the models. We would measure distances between models, test how closely those distances match a tree, and merge along that tree only when the test says it exists.
The test comes from metric geometry. The fourth-point condition states that for every three points there is a fourth point in the set (the "median") sitting exactly at the junction of the paths between them. On minimum spanning tree-like metric spaces proves that when all pairwise distances are distinct, this condition alone is equivalent to the space being a tree whose vertices are exactly the points, and that tree is the unique minimum spanning tree. Applied to a collection of models, this gives two things. It gives a yes/no answer, with a degree attached, to whether the collection branches at all. When the answer is yes, it names actual models to serve as anchors, namely the medians, so merges can be computed relative to them. Our two-sentence claim is this: a set of models fine-tuned independently from one base forms a star centered on the base, which is why flat task averaging wins and why the trees in the nowcasting paper hurt. Model collections with real lineage form genuine trees, and merging them along the recovered tree, anchored at medians, should beat flat averaging.
Core Claim
Take $n$ models $W_1,\dots,W_n$ and a distance $d$ between them. For any triple $x,y,z$, we define the median slack as the relative excess of the best in-set junction point over the perimeter bound:
$$ s(x,y,z) \;=\; \min_{p}\frac{d(p,x)+d(p,y)+d(p,z)}{\tfrac12\,[d(x,y)+d(y,z)+d(z,x)]}\;-\;1 , $$
Here $p$ ranges over the models in the set. The triangle inequality guarantees $s\ge 0$. The quantity is exactly zero when the triple has a median in the sense of A characterization of minimum spanning tree-like metric spaces. We pair this with the classical four-point slack for quadruples. That slack is the gap between the two largest of the three pairwise-sum pairings, divided by the diameter of the set. It is zero exactly for tree metrics, by Buneman's condition. The collection's tree-likeness is the 90th percentile of both slacks across all triples and quadruples.
The proposal makes three claims. The first is descriptive: independently fine-tuned task models have near-zero median slack with the base model as the median of almost every triple, so their realizing tree is a star. The second is algorithmic: in a star, the median-anchored merge reduces exactly to task arithmetic, which means adding scaled differences from the base. In a non-star tree, it becomes a recursive procedure. Each internal node $a$ (itself a real checkpoint) absorbs its children through
$$ W_a^{\text{merged}} \;=\; W_a + \textstyle\sum_{i\in \text{children}(a)} \lambda_i\,(W_i^{\text{merged}} - W_a). $$
In this update, $\lambda_i$ is a merge coefficient, set as in task arithmetic. The third claim is predictive: across many collections, the accuracy gain of tree-guided merging over flat merging should rise as tree-likeness improves, meaning as slack falls. The third claim is the falsifiable core. If the gain does not track slack, the geometry explains nothing.
Research Program
Choosing the distance. Tree tests are meaningless unless $d$ satisfies the triangle inequality. Here the metric-trade-off literature acts as a constraint on the design, not as decoration. Good Classification Measures and How to Find Them proves that no classification measure can be monotone, a true distance, and chance-corrected (constant baseline) all at once. Systematic Analysis of Cluster Similarity Indices proves the analogous result for comparing partitions. Comparing two models' predicted labels on a probe set is formally the same object as comparing a classification with a reference. So any chance-corrected disagreement score, such as one minus Cohen's kappa, must give up the distance property, and four-point tests on it are not well defined. We therefore give up the constant baseline on purpose. Our main functional distance is the correlation distance that Gösgens and coauthors single out among indices with good properties, computed on predictions over a small unlabeled probe set. We add mean total-variation distance between predicted class distributions, which is also a true metric. For a fully data-free variant, we use Euclidean distance between weight vectors. All three are metrics, so the slacks are well defined. Running a chance-corrected score as a deliberate non-metric ablation tests the theorem's practical consequence.
Recovering the tree. Real-valued distances are distinct with probability one. Under that tie-breaking condition, Hayamizu and Fukumizu's characterization says the realizing tree, if one exists, is the minimum spanning tree. Their "basic geodesic graph" construction yields it directly. The full $O(n^4)$ check is trivial for the collection sizes used in merging, where $n\le 50$. The slack converts the exact theorem into a graded diagnostic, which addresses the open problem the file notes on noise-tolerant tree-likeness. The tree also serves as a rooted schedule, rooted at the pretrained base. This connects to A Uniqueness Theorem for Clustering, which shows that single-linkage clustering is the only fixed-$k$ method that depends only on the minimum spanning tree and the ordering of distances. Our merge schedule is a single-linkage hierarchy. The new ingredients are the diagnostic that decides whether to use a hierarchy at all and the median anchors that decide what each merge is measured from.
Collections to study. We would use three families of model collections. The first is the 8 independent CLIP ViT-B/16 task fine-tunes (MNIST, DTD, EuroSAT, SUN397 and four others) used in the nowcasting paper, where we predict a star. The second is constructed lineage collections from the same 8 tasks. These are produced by sequential and branching fine-tuning: base, then SUN397, then branches to DTD and EuroSAT, and so on. We would generate 30 different random lineage trees of depth 2 to 4. They provide ground-truth topology at known, varied tree-likeness. Adding seed noise and partial training blurs the tree in a controlled way. The third is Qwen3-0.6B fine-tunes for Math, French and Code. Here we would also build a lineage variant, in which Code and Math branch from a shared reasoning fine-tune, and evaluate on GSM8K and HumanEval.
Baselines
The naive reference point is uniform weight averaging of all models. It is the method the nowcasting paper found hard to beat in many-model settings, and it costs nothing. The strongest existing data-free methods are task arithmetic and TIES. On the 28 pairwise CLIP ViT-B/16 merges, Meta-Merging by Checkpoint Nowcasting reports 90.8% average top-1 accuracy for TIES, 90.5% for task arithmetic and 89.1% for the NiNo nowcaster. On HumanEval with Qwen3-0.6B, it reports 46.3 for NiNo against 42.7 for task arithmetic. For 4-way and 8-way merges, which is where our claim lives, no reported result was found on Lacuna for task arithmetic, TIES, averaging or tree-structured NiNo. The file says only that the tree variants trail averaging. We would therefore reproduce all baselines ourselves on identical checkpoints.
Every method receives the same validation budget for choosing merge coefficients: a grid over one global $\lambda$ on the same held-out split. No method sees test data. We would also run NiNo in its hierarchical configuration, which is the system whose failure motivates the work.
Three further comparisons isolate the key ingredients. The first uses an arbitrary balanced binary tree, the kind of schedule that failed in the nowcasting paper. The second uses our recovered tree, but measures every merge from the pretrained base instead of from the median. This separates topology from anchoring. The third is an oracle that searches all rooted trees on validation accuracy, which is feasible for $n\le 8$. It gives an upper bound on what any choice of schedule can deliver. On independent fine-tunes, the bar for success is to match flat task arithmetic and TIES within noise, since our diagnostic should return a star and reduce to task arithmetic. On lineage collections, the bar is at least 1 point of average top-1 over the better of task arithmetic and TIES, while capturing most of the oracle's gain.
Why not just…?
Why not just always use flat task arithmetic or TIES? This is the real threat. If the lineage collections show no gain, the honest conclusion is that flat merging is sufficient, and the diagnostic is still useful because it explains why the nowcasting trees failed. The experiment settles the question directly. For independent fine-tunes we predict no gain. For lineage collections we predict a gain that grows as slack shrinks. Flat merging measures every difference from the base, so it double-counts shared intermediate updates when several models descend from one intermediate checkpoint. The base-anchored ablation tests exactly that mechanism.
Why not just cluster task vectors hierarchically, for instance by cosine similarity, and merge in that order? Single-linkage clustering on a metric produces exactly the minimum spanning tree hierarchy, so this alternative largely coincides with our topology. Zadeh and Ben-David's uniqueness theorem is the reason. The proposal adds two things clustering does not. It adds a slack that decides whether any hierarchy is warranted; clustering always returns one, even for a star, which is the failure the nowcasting paper observed. It also adds median anchors that are real checkpoints. A clustering run with cosine similarity, which is not a metric, is included as a baseline. If it matches us on lineage collections and does not hurt on independent ones, the median anchor is not pulling its weight.
Why not just learn the merge coefficients per model on validation data? Learned weights can compensate for a poor schedule only as far as a single global coefficient pair allows. They also give up the data-free setting that weight-space distances keep. We would run per-model coefficient search under the same validation budget, applied both to flat merging and on top of our tree. If per-model coefficients on a flat merge close the gap, topology is secondary.
Why not just search over merge orders? The search is exponential in the number of models and requires labeled validation data at every step. We run it only as an oracle for $n\le 8$, to measure how much of the achievable gain the zero-search geometric schedule recovers.
Decisive Checks
The first check is the star prediction on the 8 independent CLIP fine-tunes. With correlation distance on the probe set, the base should be the median of the large majority of the 56 triples, and median slack should be near zero. If the realizing tree is not a star, our explanation of the nowcasting result fails. The second check is topology recovery. On the 30 constructed lineage collections, the minimum spanning tree should recover the true lineage edges, scored as the fraction of correct edges, at a rate that degrades smoothly as injected noise raises slack. The third check is the central test. Across all collections, we regress the accuracy gap between tree-guided and flat merging on tree-likeness. We expect a clearly negative slope: lower slack, larger gain. A flat or positive slope falsifies the proposal.
The fourth check tests the metric constraint. Replacing correlation distance with a chance-corrected disagreement score should produce measurable triangle-inequality violations on the probe set, along with unstable, seed-dependent trees, as the impossibility theorems predict. Finally, we repeat the lineage experiment on Qwen3-0.6B using GSM8K and HumanEval, to test whether the effect carries over from vision classifiers to generative language models. In that setting, the reference point for the task arithmetic baseline is the reported 42.7 on HumanEval.
References
- Meta-Merging by Checkpoint Nowcasting
- On minimum spanning tree-like metric spaces
- A characterization of minimum spanning tree-like metric spaces
- Good Classification Measures and How to Find Them
- Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures
- A Uniqueness Theorem for Clustering