What Does Cross-Model Coordination Cost in Secure Model Merging?

Merging combines several separately fine-tuned copies of one base network into a single multi-task model by combining their weight changes. The best merging recipes do more than add those changes. TIES-Merging, for example, first holds a per-coordinate "vote" across all contributing models on which sign each weight change should take, then averages only the changes that agree. Secure aggregation is the cryptographic and information-theoretic toolkit that lets a server learn the sum of users' vectors and nothing else. It was built for plain sums. A vote that needs every model's signs, followed by an average whose denominator depends on how many models agreed at each coordinate, is not a sum. So today there is no principled way to run the stronger merging recipes when the contributing models are confidential. Nobody knows what running them would cost in extra communication, extra shared keys, or extra information revealed to the server.

This note proposes to measure that price. We first split merge operators into two groups. "Separable" operators, such as local sparsification or masking, let each owner preprocess its own weight change so that the server only ever needs a sum. "Coordinated" operators, such as sign election and disjoint averaging, need information across owners. For the coordinated group, we then derive an achievable protocol (an inner bound) and a converse (an outer bound) on communication rate, key size, and leakage. Finally, we test empirically whether coordination buys any accuracy that separable operators cannot. The two possible answers are both useful. If coordination adds nothing, private merging should use separable operators and plain secure aggregation. If it adds something, we get the first protocol that pays for it at a characterized cost.

Core Claim

Our central claim has three parts. The first is that every interference-resolving merge can be written as a composition of a small number of secure sums, interleaved with public broadcasts. The second is that the number of rounds, and what is revealed between them, sets a lower bound on key size and leakage. The third is that this bound can be computed for the specific statistics TIES-style merging needs.

To make "leakage" precise, let $\tau_1,\dots,\tau_K$ be the $K$ owners' task vectors (weight changes relative to the base model), quantized to a finite field. Let $f(\tau_1,\dots,\tau_K)$ be the merged model. Let $M$ be everything the server observes during the protocol. We define the excess leakage as

$$ L = I(\tau_1,\dots,\tau_K ; M \mid f(\tau_1,\dots,\tau_K)), $$

the mutual information between the private inputs and the server's view, beyond what the output itself reveals. Plain secure aggregation of a sum achieves $L=0$ when $f$ is the sum. For coordinated merges we ask a concrete question: what is the smallest achievable pair of communication rate and key size, given a target $L$ and tolerance to $K-U$ owners dropping out, where $U$ is the number of owners who remain until the end?

Two strands of prior work connect here, and they have never been joined.

The first strand is secure-aggregation capacity theory. On the Information Theoretic Secure Aggregation With Uncoded Groupwise Keys supplies our method. It characterizes the rate needed for a server to learn a sum over a finite field when keys are shared within groups of size $S$. The rate is exactly optimal when $S > K-U$ and is provably suboptimal otherwise. That paper treats model updates as abstract symbols whose only fate is to be summed, so it says nothing about interference.

The second strand is merging. TIES-Merging: Resolving Interference When Merging Models motivates the problem, since its gains come precisely from cross-model sign election. Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks and Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch constrain it. Their masking and dropping steps are applied to each task vector locally, which makes them separable and therefore already compatible with plain secure aggregation.

The link we exploit is structural. Once each owner's changes are reduced to their signs (ternary values), a majority-count sign election becomes a secure sum of those ternary vectors. Disjoint averaging becomes two further secure sums, one of the agreeing values and one of the agreement counts. These are computed after the server publicly broadcasts the elected signs. Each extra round consumes fresh keys. This is exactly where the unresolved $S \le K-U$ regime of the groupwise-key paper becomes the binding constraint.

Research Program

The program has three parts.

The first part is theory. We will write sign-elected disjoint merging as a two-round secure computation. In round one, owners securely sum their trimmed sign vectors. In round two, owners that agree with the elected sign securely sum their values and indicator counts. The server learns the vote tallies and the per-coordinate counts in addition to the merged output. This gives an inner bound with explicit rate, key size, and leakage $L$.

We will then prove converses of three kinds:

  • a lower bound on total key size for any two-round protocol that tolerates $K-U$ dropouts;
  • a lower bound on the leakage of any one-round protocol computing the same output, showing that one round cannot achieve small $L$ without generic multiparty computation;
  • a characterization of how much of $L$ is forced by the output itself.

The output is a ratio of sum to count, so it hides the counts only partially, and quantifying this is a genuine open question. We will also study a leakage-reducing variant. In it the counts are never revealed. Instead, owners pre-scale their values using a publicly broadcast tally, which trades the count leakage for extra rate. We expect the inner-outer gap to be closed for $S > K-U$ and to stay open for $S \le K-U$, where the plain-sum problem itself is already unresolved.

The second part is an honest accuracy test of whether coordination matters. An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse constrains us directly. Across 64 checkpoints it reports near-zero correlation between sign-conflict metrics and merge failure. It also proves a distortion floor for any convex merge, set by the diameter $\Delta$ of the tasks' representation clusters in hidden-state space. If sign conflicts do not predict failure, the accuracy value of coordinated sign election may be small once a strong separable operator is used. We therefore treat "coordination gain over the best separable merge" as the quantity to measure, not an assumption. We will also test whether the gain concentrates on task pairs with small representation divergence, where a parameter-space fix can still act, rather than on pairs with large divergence.

The third part is empirical leakage. The server's intermediate view consists of the vote tallies and counts. We will attack this view directly: we try to infer which owner holds which task, and we try to reconstruct individual task vectors. We measure how much the theoretical $L$ matters in practice.

Baselines

The do-nothing reference is plain secure aggregation of the raw task-vector sum, which amounts to task arithmetic under the uncoded groupwise-key protocol of On the Information Theoretic Secure Aggregation With Uncoded Groupwise Keys. It has zero excess leakage and the known optimal rate for $S > K-U$. It sets the floor for accuracy and the ceiling for efficiency.

The strongest competitor is the best separable operator run through the same secure sum. This is Breadcrumbs' two-sided masking, which drops both the smallest and the largest changes in each layer, applied locally before aggregation. It inherits zero excess leakage and optimal rate for free, so it is the method our protocol must beat on accuracy. The file reports that Breadcrumbs improves accuracy over task arithmetic by 5.7% on the 8-task vision benchmark (Model Breadcrumbs). No reported absolute Breadcrumbs or TIES accuracy on that benchmark was found on Lacuna, so we will reproduce both in plaintext first.

The plaintext upper reference is TIES-Merging run with no privacy. The file reports gains of 1.8% and 1.5% average accuracy on ViT models, and 2.3% and 1.7% in-domain on T5-base and T5-large (TIES-Merging). Our secure protocol should match plaintext TIES up to quantization error.

For the cost comparison, we run the same TIES computation under generic secure computation, meaning coded-key secure aggregation plus secret-shared comparison. The file reports no number for TIES under such schemes. For plain sums it reports that uncoded groupwise keys cut key-sharing time by up to 31.7% relative to LightSecAgg and aggregation time by up to 53% relative to SecAgg on Amazon EC2 (On the Information Theoretic Secure Aggregation With Uncoded Groupwise Keys).

One ablation isolates the key ingredient. It keeps the round-two disjoint average but replaces the elected sign with each owner's own sign. This removes cross-owner coordination while keeping everything else fixed.

For fairness, all methods use the same fine-tuned checkpoints and the same fixed-point quantization to the finite field. Each gets the same hyperparameter search over trim fraction, mask bands, and scaling coefficient, using the same validation split. Beating the separable baseline means one of two things. Either the average accuracy gain is larger than the seed-to-seed spread on at least two of the three benchmarks, or there is a consistent gain on the high-interference task subsets.

Why not just…?

Why not just use Breadcrumbs or DARE-style local masking with plain secure aggregation? This is the real threat, and the program is built around it. These operators are separable, so they are already private at optimal rate. The collapse study's finding that sign conflicts barely predict failure (An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse) suggests coordination may add little. The coordination-gain experiment settles this directly. If the gain is within noise, we report that result and give the practical recommendation: use separable operators for private merging. The theory then serves to show that coordinated operators cost strictly more.

Why not just apply differential privacy instead? Differential privacy adds calibrated noise to guarantee that no single training example changes the output much. It protects training examples from inference through the released model. It does not hide one owner's task vector from the server during merging, which is the threat we address. It also hurts mergeability. When Privacy Hurts Mergeability reports that DP fine-tuning pushes models into sharp regions of the loss landscape and away from the base weights. Its geometry-aware fix reaches 77.0 average GLUE against 74.7 for naive DP merging on RoBERTa-Large. The two guarantees compose, and we will report our protocol applied on top of DP-Merging checkpoints.

Why not just use generic secure multiparty computation, which can compute TIES exactly? It can. Secure comparison over billions of coordinates, with dropouts, is precisely the overhead the groupwise-key line of work was designed to avoid. Our protocol avoids comparison altogether by using majority-count sign election, which is a sum. The runtime baseline above measures the difference. The converse also tells us when no cheaper protocol can exist.

Proposed Experiments

We will use three benchmarks from the file:

  • the 8-task CLIP vision benchmark (SUN397, Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SVHN) on ViT-B/32 and ViT-L/14;
  • the T5 multi-task NLP suite from TIES-Merging;
  • GLUE on RoBERTa-Large, including WNLI and MRPC, which the collapse study flags as tasks that cause merges to fail.

The 200-task Breadcrumbs benchmark provides a large $K$. This matters for testing dropouts and the $S \le K-U$ regime, because $K=8$ is too small to separate the regimes meaningfully.

We will run four decisive checks. The first confirms that the secure two-round protocol reproduces plaintext majority-count TIES to within quantization error. If it does not, the construction is wrong. The second measures coordination gain over secure Breadcrumbs, both on each benchmark and stratified by the collapse study's Merging Difficulty Score, a per-task-pair score for how hard the pair is to merge. We predict that any gain appears on intermediate-difficulty pairs and vanishes on high-divergence pairs, where no parameter-space operator can help. The third plots measured key size and per-owner communication against our converse across dropout rates $K-U$ and group sizes $S$, and reports the multiplicative gap in each regime. The fourth attacks the revealed vote tallies and counts to infer task identity and reconstruct task vectors, measured by cosine similarity. It compares the plain protocol with the count-hiding variant, which tells us whether the theoretical leakage term is exploitable in practice.