Cycle Residuals as a Data-Free Veto for Merging Small-Cohort Clinical Models

When hospitals that each hold only a few dozen patients want one shared model, the usual approach is to average their networks' weights. Averaging is a merge, and a merge is hard to take back. If one site's model has drifted into an incompatible solution, averaging it in damages the shared model for everyone. Checking each merge on held-out patients is the obvious safeguard, but a site with 30 patients cannot spare a validation set large enough to tell a good merge from a bad one. We propose a check that uses no data at all. Take three site models, match the hidden units of A to B, B to C, and C back to A, and count how many units fail to return to where they started. This count is the cycle residual. When the three models sit in a compatible region of weight space, the matchings are well determined and compose to the identity. When one model is incompatible, its matchings are close to arbitrary and the loop breaks.

The core claim is that this triangle residual is a reliable veto on irreversible merges. We use it inside an agglomerative procedure, meaning one that repeatedly fuses the two closest groups. The procedure groups site models bottom-up and merges two groups only if every triangle spanning them closes. It can also undo a merge after the fact by moving a model to a different group. The result is a small hierarchy of merged models instead of one forced global average. It isolates divergent sites without corrupting the rest, and the decision to merge never depends on how many patients a site has.

What is new relative to the closest work

The closest work is $C^2M^3$: Cycle-Consistent Multi-Model Merging. It maps all $n$ models to one shared "universe" ordering of units, so cycle consistency holds by construction, and it solves the permutations of all layers jointly with Frank-Wolfe. That paper supplies our matching machinery, but it uses cycle consistency in the opposite way. Because consistency is guaranteed by construction, it can no longer serve as a diagnostic. The method is also flat, and its own authors identify the "hardest pair" problem (companion version): one divergent model degrades the alignment of all the others.

Deep Neural Network Fusion via Graph Matching (GAMF) adds second-order channel matching and multi-model cycle consistency for federated fusion. It also enforces consistency rather than measuring it, and it aggregates in a single flat step. We deliberately keep pairwise matchings unsynchronized, because their failure to close is the signal we want.

Our merge-control design comes from the work on irreversible merging. PuzzleFusion++ shows that putting a verifier in front of agglomerative merges prevents cascading errors in 3D fracture assembly. Its verifier, however, is a learned pairwise scorer trained on abundant data, and it never checks loops. Grinch supplies local rotate and graft operations that repair a hierarchy built from incomplete information. We reuse graft as our undo step.

HC-SMoE is the cautionary example. It merges expert networks by average-linkage clustering on calibration outputs and has no undo, so a bad early merge is permanent. The cascading-error analysis in Clustering-based Inference for Biomedical Entity Linking explains why this matters: one wrong merge shifts the group centroid and pulls later members into the corrupted group.

The clinical motivation comes from Federated learning enables big data for rare cancer boundary detection. It shows that federation across 71 sites helps a great deal, but it aggregates by plain coordinate-wise averaging with no alignment and no verification. The setting is also constrained by the observation of Bria et al. (Towards AI-driven Next Generation Personalized Healthcare and Well-being) that many clinical deployments have fewer than 100 patients per institution. Under that constraint, any verifier that needs held-out data is starved.

Core Claim

For site models $i,j,k$ with $L$ aligned layers, let $P^{l}_{ij}$ be the permutation that weight matching assigns from model $i$'s units in layer $l$ to model $j$'s, and let $d_l$ be the width of that layer. The triangle residual is

$$r_{ijk} = \frac{1}{L}\sum_{l=1}^{L}\left(1 - \frac{1}{d_l}\,\mathrm{tr}\!\left(P^{l}_{ki}P^{l}_{jk}P^{l}_{ij}\right)\right).$$

This is the average fraction of units that do not return to themselves after going around the loop. It is 0 for a perfectly consistent triple and about $1 - 1/d_l$ per layer for random permutations.

Two groups $G$ and $H$ are allowed to merge only if

$$\max_{i \in G,\; j \in H,\; k \in G \cup H} r_{ijk} \le \tau,$$

where $\tau$ is a threshold on the worst triangle that spans the two groups.

The intuition for why this beats pairwise matching cost runs as follows. Weight matching always returns some permutation. Between models in incompatible basins, many permutations score about equally well, so the one returned is close to arbitrary, and arbitrary choices do not compose around a loop. The pairwise matching cost, by contrast, mixes basin mismatch with nuisance factors such as weight scale and sparsity, and these differ across sites trained on different scanners.

The residual depends only on weights. Its reliability therefore does not shrink with local cohort size. Any held-out estimate of merge quality, in contrast, has a standard error that grows like $1/\sqrt{n}$ as the number of validation patients $n$ falls.

Research Program

We will build a hierarchical merging procedure for site models in three steps.

First, we compute pairwise per-layer matchings between all site models using the matching objective of $C^2M^3$, solved pairwise rather than through the universe. For the tens of sites typical of a clinical consortium, this means $\binom{n}{2}$ matchings and $\binom{n}{3}$ residuals, which is cheap relative to training.

Second, we run agglomerative grouping. The linkage between two groups is their worst spanning triangle residual, and merges stop at $\tau$. Within each accepted group we merge with the $C^2M^3$ universe and REPAIR activation renormalization, so that the inside of a group gets the consistency-by-construction benefit.

Third, after each merge we run a Grinch-style graft pass. Any model whose residuals with its own group exceed its residuals with another group is moved to that group, and the affected groups are re-merged. This is the undo step that HC-SMoE lacks.

Choosing $\tau$ is the main brittleness risk; PuzzleFusion++ notes the same problem for its 0.90 threshold. We will compare three choices. The first is the largest gap in the sorted residual distribution, which needs no data. The second is a null calibrated by retraining one site's model under small weight perturbations, which reuses that site's own data and requires no validation split. The third is a fixed value. We will report sensitivity curves for all three rather than a single tuned number.

On the analysis side, we will test the identifiability intuition directly. We will measure how the residual changes as two models move from a shared initialization to independent initializations, and as their training distributions move from identical to disjoint. The aim is to establish when the residual reflects basin structure and when it reflects only matching noise.

Baselines

The naive reference points are plain coordinate-wise averaging, as used in the glioblastoma federation, and local-only training, where each site keeps its own model. Averaging is what clinical consortia actually deploy. Local-only shows whether merging helps a site at all.

The strongest existing method is flat $C^2M^3$ with REPAIR across all sites. On a 5-model ResNet-20 merge on CIFAR-100, it reports 69% accuracy against 50% for MergeMany ($C^2M^3$: Cycle-Consistent Multi-Model Merging). We also include GAMF, which reports 49.80% on 5-client federated CIFAR-100, compared with 47.50% for FedMA and 1.00% for OTFusion (Deep Neural Network Fusion via Graph Matching with Applications to Model Ensemble and Federated Learning). All methods will use the same site models, the same matching solver where one applies, the same REPAIR calibration images (a fixed small number per site), and equal wall-clock budgets. Any method that has a threshold or group count gets the same tuning procedure.

We will reproduce the 69% figure before adding our stresses. Beating $C^2M^3$ means at least matching its mean accuracy or Dice when all sites are compatible, and showing a clear gain on clean-site performance when divergent sites are present.

Three ablations isolate the key ingredient. The first, and most important, replaces the triangle residual with the pairwise matching cost as the linkage, keeping everything else identical. The second uses a held-out-loss verifier in the style of PuzzleFusion++, in which a merge is accepted if it does not raise loss on validation patients drawn from each site's small cohort. The third removes the graft pass. An oracle that knows the true site groups serves as the upper bound.

Why not just…?

Why not just use the pairwise matching cost as the linkage? This is the real threat. If pairwise cost separates compatible from incompatible sites as well as triangles do, the cycle machinery is unnecessary. Our argument is that pairwise cost is confounded by weight scale and per-site training differences. That argument is a hypothesis, not evidence from the file, and the first ablation settles it directly. Across thousands of model triples we will compute the rank correlation between each signal and the accuracy drop observed after merging, and we will compare how well each recovers the true groups. If pairwise cost matches the residual, we will report that and reduce the proposal to the graft-based undo, using pairwise linkage.

Why not just run flat $C^2M^3$ and accept the hardest-pair cost? When all sites are compatible, flat $C^2M^3$ is probably as good as anything, and we expect to tie it. Its own authors, however, report that one divergent model degrades the whole collection. Our divergent-site injection experiment measures exactly what that degradation costs the clean sites, and whether our method avoids it.

Why not just verify each merge on a small validation set, as PuzzleFusion++ verifies with a learned scorer? With 20 to 50 patients per site, held-out Dice estimates are noisy, so accept/reject decisions become close to coin flips near the threshold. This shortfall is our prediction, not a result from the file. The sample-size sweep below tests it. If the data-based verifier stays reliable down to small cohorts, it is a simpler and better choice than ours.

Why not just downweight outlier updates during averaging? Downweighting an outlier still forces a single shared model and discards the outlier site's knowledge. It also leaves the permutation mismatch in place, which is the failure that GAMF's vanilla-averaging baseline exposes (60.34% versus 79.40% after alignment on CIFAR-10). We include a variant of $C^2M^3$ that drops the model with the highest alignment cost, so this alternative is tested directly.

Proposed Experiments

The controlled benchmark is CIFAR-100 with ResNet-20, following $C^2M^3$. Clients come from three latent groups with disjoint class-distribution skews, plus zero, one, or two deliberately divergent clients trained on extreme non-IID splits. Clients hold between 50 and 5,000 images each. We measure accuracy of the merged model on each client's test distribution, worst-client accuracy, the adjusted Rand index between recovered groups and true groups, and the change in clean-client accuracy when divergent clients are added.

The clinical benchmarks are retinal fundus optic disc and cup segmentation on the five public sites used by TTDG-MGM (Test-Time Domain Generalization via Universe Learning: A Multi-Graph Matching Approach for Medical Image Segmentation) and polyp segmentation on its four sites (BKAI-IGH, CVC-ClinicDB, ETIS, Kvasir). Each site is subsampled to 20, 50, or 100 training images to mimic small cohorts. Each site trains a U-Net from a shared initialization for many local epochs, so that drift is real. We report Dice on each site's held-out test images and on a left-out site.

No reported result for multi-model merging on these sites was found on Lacuna. The 88.46% Dice that TTDG-MGM reports comes from test-time adaptation of a single model, against 69.46% with no adaptation. We cite it only as context, not as a directly comparable baseline. TTDG-MGM's polyp results are not given numerically in the file.

Three checks decide the proposal. The first is predictive: across all triples, does the cycle residual predict post-merge damage better than pairwise cost, as measured by rank correlation and by precision of the veto? The second is protective: with a divergent site injected, does our method keep clean-site performance within noise of the oracle grouping while flat $C^2M^3$ drops? The third is scarcity: as validation cohorts shrink from 100 to 10 patients, does the accuracy of the held-out-loss verifier's decisions fall while the residual veto's decisions stay fixed?

If the residual wins on the first check and the scarcity check, the claim stands. If pairwise cost ties it on the first check, the contribution reduces to hierarchical merging with undo. If neither beats flat $C^2M^3$ on the protective check, then grouping does not help in these regimes, and we will report that as a negative result.