Model merging combines several copies of one pretrained network, each fine-tuned for a different skill such as math, code, or instruction following, into a single set of weights without retraining on pooled data. Current methods do this in one shot. They take the finished experts, compute a weighted average of their weights, and stop. The best of them weight each parameter by how "stiff" it is for each expert: how sharply that expert's loss rises when the parameter moves, a quantity called curvature. A one-shot average cannot fix the case where two experts have settled in places that are each low-loss for their own task but sit far apart. In that case any compromise lands on a steep wall for at least one of them. We propose to borrow the structure of the Delphi method, a protocol for building consensus among human experts in which panelists answer independently, see only an anonymized statistical summary of the group, and then either move toward the group or defend their position. Here the "panelists" are fine-tuned models. In each round, every expert sees only the current curvature-weighted consensus. It moves toward that consensus along directions where its own loss is flat, which costs it almost nothing, and it holds firm along directions where its loss is steep. The firmness it reports then becomes the weight the facilitator gives its position in the next aggregate.
The claim is that a few such rounds, using only local data and a small gradient budget, recover much of the gap between one-shot curvature-aware merging and multitask training. We also claim the gain has a measurable geometric cause: the experts' low-curvature directions come into closer alignment from round to round. This is new relative to existing curvature-aware merging, which never lets the experts move, and relative to plain multi-round federated averaging, which pulls every parameter toward the mean equally, regardless of whether conceding there is cheap or ruinous.
Why this gap is real
Three merging papers supply the method and define the bar. The Optimization Trajectory Aware merging method in Harnessing Optimization Dynamics for Curvature-Informed Model Merging reuses Adam's running average of squared gradients as a free estimate of each parameter's curvature. It keeps only high-curvature parameter updates, weights each expert's contribution by stiffness, and reports that independently fine-tuned experts share similar curvature profiles. CAMEx: Curvature-aware Merging of Experts adds a learned curvature matrix, compressed into a cheap factored form, to merging in a centrally trained setup. MAP fits a quadratic surrogate of task metrics to search cheaply over the scalar coefficients that mix the experts. All three treat the experts as frozen vectors. We reuse the first paper's curvature estimate as our stiffness signal and add the step they lack: letting experts re-optimize in response to the consensus.
The motivating tension comes from Beyond Spectra: Eigenvector Overlaps in Loss Geometry. It shows that two curvature operators with identical eigenvalues can have rotated eigenvectors, so flat directions of one land on steep directions of the other. That paper studies train versus test curvature of a single model, but the logic carries over to two experts. Matching curvature magnitudes, which is what the Shared Geometry Hypothesis checks, does not imply matching flat directions. A one-shot average has no way to repair the rotation, but an expert that moves inside its own flat subspace can. A loss curvature account of fine-tuning fragility adds a second motivation and a test. Fine-tuning on concentrated data leaves models in narrow, high-curvature "wrappers" that collapse when other data returns. The paper reports roughly 4x less relative degradation at concentration 0.25 than at 1.0 on its chemistry task. A merge whose experts have negotiated toward one another should inherit less of that fragility, and this is checkable directly.
On the Delphi side, Voting or Consensus? Decision-Making in Multi-Agent Debate shows two things that act as constraints on our design. Independent first drafts help, with a reported 3.3% average gain from blinded drafting. More rounds of exchange can also cause "drift," in which quality degrades as agents lose grounding. We take both seriously. Experts start from their own fine-tuned solution, never from a peer's, and we treat the round count as a quantity to measure rather than assume that more is better. Classical Delphi studies such as VICCCS contribute the termination idea, which is to stop when the summary statistics stabilize across rounds.
Core Mechanism
Let $\theta_k$ be the weights of expert $k$ out of $K$, and let $F_k$ be its diagonal curvature estimate (the Adam second-moment proxy, refreshed after each round). The facilitator forms a consensus $\bar\theta$ by curvature-weighted averaging per coordinate, $\bar\theta = (\sum_k F_k)^{-1}\sum_k F_k \theta_k$. This rule matches the one-shot method in round one, so round one is the strongest existing baseline.
Each expert then runs a short local update on its own data only:
$$\min_{\theta}\; \mathcal{L}_k(\theta) + \frac{\gamma}{2}\,(\theta-\bar\theta)^\top P_k\,(\theta-\bar\theta), \qquad P_k = \mathrm{diag}\!\left(\frac{\bar f_k}{F_k+\epsilon}\right).$$
Here $\mathcal{L}_k$ is expert $k$'s task loss, $\gamma$ is the pull strength, $\bar f_k$ is the mean of $F_k$ (so $P_k$ has unit scale), and $\epsilon$ is a small floor. The pull toward consensus is strong where the expert is flat ("concede") and weak where it is stiff ("defend"). After the update, the expert's refreshed $F_k$ is its anonymized justification: coordinates it defended stay stiff and earn more weight in the next aggregate. Experts never see one another's weights, only $\bar\theta$, which mirrors Delphi's anonymity and controlled feedback.
The protocol stops when the relative change in $\bar\theta$ and in the curvature-weighted dispersion of the experts around it both fall below a preset threshold, or at a hard cap of five rounds. The output is the final $\bar\theta$.
Research Program
The work has three parts. The first builds the protocol on top of the released merging pipelines and holds the total number of gradient steps fixed across all compared methods. The second measures geometry round by round. After each round we estimate the pairwise overlap between experts' low-curvature subspaces using the Overlap-KPM estimator from Beyond Spectra, restricted to sampled layers to keep cost manageable. We test whether rising overlap predicts the merged model's accuracy better than curvature magnitudes alone do. The third tests fragility. Following A loss curvature account of fine-tuning fragility, we measure the merged model's directional curvature along the reversion path back toward base-model data, and its relative degradation when training on base data resumes.
Baselines
The naive reference point is uniform weight averaging of the experts. On the Tulu-3 supervised fine-tuning mixture evaluated through the OLMES toolkit, Harnessing Optimization Dynamics for Curvature-Informed Model Merging reports an average score of 0.551 for linear averaging and 0.565 for TIES, a heuristic that trims small updates and resolves sign conflicts. We rerun both on our exact experts.
The strongest existing method is one-shot curvature-aware merging (OTA merging), with a reported Tulu-3 average of 0.582 in the same paper. It is the right main competitor because it already exploits the same curvature signal we use, so any gain we show must come from iteration and the concede-or-defend rule rather than from curvature information as such. It is also exactly our round one, which makes the comparison clean. Beating it means exceeding its rerun score on our experts by more than the spread across three seeds, at matched total compute. We count the extra local steps against our method and give OTA the same steps as a post-merge budget, described next.
CAMEx serves as a second curvature-aware reference where its centralized setup is feasible. The file reports its gains on GLUE CoLA (60.06 vs 58.54 for TIES) but has no Tulu-3 figure.
Two ablations isolate the key ingredient. The first is isotropic multi-round averaging: the same rounds, local steps, and data, but with $P_k$ set to the identity, as in proximal federated training. The second is our protocol with the stiffness-based aggregate replaced by a plain mean. All methods share the same experts, the same per-expert data, the same total gradient steps, and the same tuning budget for $\gamma$ and the learning rate, selected on a held-out split.
We also report multitask fine-tuning on pooled data as an upper bound that violates the no-pooling constraint. It shows how much of the remaining gap the protocol closes.
Why not just…?
Why not just run one-shot curvature merging and stop? Because it cannot repair rotated flat subspaces. Beyond Spectra shows that matching eigenvalues do not imply matching eigenvectors, and the Shared Geometry evidence checks only magnitudes. If round-by-round overlap does not rise, or rises without accuracy gains, this objection wins and we will say so.
Why not just do multi-round federated averaging with a proximal pull? This is the real threat, since our method is that procedure with a curvature-shaped pull and a curvature-weighted aggregate. The isotropic ablation settles it directly. If the identity-pull variant matches ours at equal steps, the Delphi structure adds nothing beyond iteration, and the contribution reduces to the geometric measurements.
Why not just fine-tune the one-shot merge briefly on each expert's data in turn? Sequential fine-tuning is exactly the regime that A loss curvature account of fine-tuning fragility shows erases earlier skills through high directional curvature. We nonetheless run it at matched steps. We expect it to favor whichever task comes last and to show higher reversion curvature than our merge.
Why not just search the mixing coefficients with MAP? MAP cuts the search from 2,500 to 60 evaluations, but it moves only per-task scalars, so it cannot change the experts' internal geometry. We include MAP-tuned coefficients as a baseline on the vision suite, where its code runs as released.
Proposed Experiments
The main language experiment uses the Tulu-3 mixture and the OTA expert split (math, code, instruction following), scored with OLMES against the 0.582 reference. The vision experiment merges CLIP ViT-B/32 experts on MNIST, SUN397, SVHN, and EuroSAT using the MAP codebase. The file has no reported per-task accuracy for this suite (no reported result found on Lacuna for merged ViT-B/32 accuracy), so all numbers there are our own reruns. The fragility experiment builds experts fine-tuned on chemistry, music, and biomedical text at concentration 1.0, merges them, and resumes training on base data.
Three outcomes are decisive. First, the protocol must beat both one-shot OTA and the isotropic multi-round ablation at equal compute. Second, the gain must track the measured increase in flat-subspace overlap, not merely the number of rounds. Third, accuracy as a function of round count must reveal whether a drift peak exists. The multi-agent debate results predict one, so we expect gains to saturate within two to three rounds. A merge that improves accuracy while overlap stays flat, or that keeps improving well past five rounds, would contradict the proposed mechanism.
References
- Harnessing Optimization Dynamics for Curvature-Informed Model Merging
- CAMEx: Curvature-aware Merging of Experts
- MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
- Beyond Spectra: Eigenvector Overlaps in Loss Geometry
- A loss curvature account of fine-tuning fragility
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- The Vascular Impairment of Cognition Classification Consensus Study