When knowledge moves between neural networks of different families or sizes, for example from a large LLaMA model into a small Qwen or Typhoon model, weights cannot simply be averaged, because the two networks have different shapes and no shared starting point. The most direct recent method, Transport and Merge, gets around this with optimal transport, a way of matching neurons in one model to neurons in the other using their activations on about 2,000 calibration examples. It then overwrites only the $k$ most active target neurons with the mapped donor weights, and finally repairs the damage with light fine-tuning. The step that decides which $k$ neurons to overwrite is a sparse selection problem, and the paper treats it as a hyperparameter with a one-shot heuristic.
We propose to treat neuron transplantation as a sparse regression problem at each layer, and to solve it with hard thresholding pursuit. This is a family of algorithms from sparse signal recovery that alternate three steps: a gradient step, keeping only the top $k$ entries, and re-fitting the kept entries exactly (called debiasing). The claim is practical. Iterating selection with a closed-form least-squares refit should (i) pick better neuron sets than one-shot top-$k$ when the candidate neurons have correlated activations, and (ii) remove most of the need for post-merge fine-tuning, because the debiasing step is itself a closed-form repair. Sparse recovery theory also supplies a measurable quantity, the restricted condition number of the calibration activation matrix. It predicts when pursuit should beat top-$k$, which turns a theory import into a falsifiable claim.
Core Claim
The layer-wise transplant problem is chosen deliberately. If one selects which output rows of a single weight matrix to overwrite, the least-squares objective separates row by row. Greedy top-$k$ is then exactly optimal, and pursuit buys nothing. We therefore select hidden neurons of a feed-forward block. Consider a transformer MLP with hidden activations $H \in \mathbb{R}^{n \times m}$ on $n$ calibration tokens and $m$ hidden neurons, and down-projection $W \in \mathbb{R}^{m \times d}$. Transplanting neuron $j$ replaces column $j$ of $H$ with the donor-mapped activation and row $j$ of $W$ with the mapped donor row. Let $Z_d$ be the donor block's output mapped into target coordinates on donor-skill data, and $Z_t$ the original target output on target-domain data. We solve
$$\min_{S \subseteq [m],\ |S| \le k,\ W} \; \|H_S^{d} W - Z_d\|_F^2 + \lambda \|H_S^{t} W - Z_t\|_F^2,$$
where $H_S^{d}$ and $H_S^{t}$ are the hidden activations on donor and target data after transplanting the neurons in $S$, and $\lambda$ trades new skill against retained ability. Neurons interact through the Gram matrix of $H$, so selection is genuinely coupled. For a fixed $S$, re-fitting $W$ is an ordinary least-squares solve on a $m \times m$ system (a few thousand dimensions per layer). This is cheap, and it sidesteps the known problem that exact debiasing in Exact Recovery of Hard Thresholding Pursuit becomes intractable over millions of raw weights.
Theory enters in three concrete ways. First, Exact Recovery of Hard Thresholding Pursuit shows that debiased pursuit recovers the right support even under poor conditioning if it searches at a slightly larger sparsity than the true one. This gives an explicit rule: select at $k > \bar{k}$ and prune back. Second, A Tight Bound of Hard Thresholding shows that the error amplification of top-$k$ truncation is at most the golden ratio, about 1.618, rather than 2, and shrinks as the ratio of kept to true sparsity grows. We will derive the analogous statement for selecting whole neurons by their contribution norm. We expect it to carry over, because group selection is thresholding on the vector of group norms, but this is a lemma to prove, not an assumption. Third, Orthogonal Matching Pursuit with Replacement shows that swapping the weakest selected coordinate for the strongest unselected one fixes irrevocable greedy mistakes. We use a replacement variant as a cheaper alternative to full pursuit.
We do not claim the textbook conditions hold for real activations. The restricted condition number bound of about 1.14 from the pursuit paper will almost certainly be violated. The contribution is to measure that quantity per layer and test whether it predicts the gap between pursuit and top-$k$.
Research Program
We build three components. The first is the alignment front end, which we reuse from Transport and Merge: optimal-transport maps computed from the same 2,000-sample calibration budget. Keeping it fixed isolates the selection step. The second is the selection back end. It comes in three variants that share one closed-form refit: one-shot top-$k$ by activation, one-shot top-$k$ followed by a single debiasing solve, and full debiased pursuit with over-sparsity and optional replacement moves. The third is a diagnostic that computes, per layer, the restricted condition number of the calibration hidden-activation matrix on candidate supports of size $2k$, using random-support sampling as an estimate.
Theoretically, we will state the group version of the tight thresholding bound and the resulting per-layer error bound on the transplanted block output. We will then propagate it loosely through depth as a sanity bound, not a headline guarantee.
We also pre-register a vision testbed where ground truth is controllable. The testbed is width-mismatched networks within a family, as in Training-free Heterogeneous Model Merging, whose Elastic Neuron Zipping merges the most similar neuron pairs greedily until the widths match. That greedy zipping is exactly the one-shot selection we argue is fragile under correlation, so it is the natural foil.
Baselines
The naive reference point is the unmodified target model. On the file's LLM setting, its reported numbers are CMMLU (Cantonese) 25.26 and MGSM (Thai) 0.50, from Transport and Merge. A second naive reference is random selection of $k$ neurons followed by the same refit. It plays the role that DARE's random dropping, from Language Models are Super Mario, plays in homologous merging, because DARE itself cannot run without a shared base.
The strongest existing method is full Transport and Merge: optimal-transport alignment, top-$k$ active-neuron replacement, and residual-frozen fine-tuning. It is the only cited method that actually crosses model families at LLM scale. Its reported gains are CMMLU rising from 25.26 to 27.44, MGSM Thai from 0.50 to 0.72, and MalayMMLU +6.51% in Humanities and +5.99% in Social Sciences. We will run it from the same donor/target pairs with the same calibration samples and report both our reproduction and the paper's numbers.
We compare in two regimes. In the training-free regime, our method with no fine-tuning faces Transport and Merge with its fine-tuning stage removed. In the matched-compute regime, both receive the same fine-tuning token budget after selection. Beating Transport and Merge means one of two things. Either our training-free variant reaches its fine-tuned score within noise, for example at least 27.4 on CMMLU. Or, at matched fine-tuning budget, we exceed it across the three cross-lingual benchmarks over several seeds.
On the vision testbed, the strongest competitor is HeteroZ-LMA from Training-free Heterogeneous Model Merging. It reports about 56% joint accuracy for ResNet-26 plus ResNet-50 on CIFAR-100 and 32.66% multi-task joint accuracy on ImageNet. We swap its zipping step for our selection while keeping its depth alignment, so the comparison changes one component.
The key ablation isolates the ingredient: one-shot top-$k$ with a single debiasing solve, versus the same with iterated pursuit and over-sparsity. If they tie, the contribution is the refit, not the pursuit, and we will report it that way.
Why not just…?
Why not just keep top-$k$ and add one least-squares refit? This is the real threat, since much of the gain may come from debiasing alone. In theory, a single refit cannot correct a wrong support. Pursuit re-selects after refitting, and replacement in Orthogonal Matching Pursuit with Replacement exists precisely to undo early mistakes. Whether those mistakes happen in practice depends on activation correlation. The ablation above settles it, and the condition-number diagnostic predicts where the two should diverge. If they never diverge, we fall back to the simpler method and report the refit as the finding.
Why not just use a smarter one-shot score, such as the pathway-aware saliency of Saliency-Aware Model Merging? That method beats magnitude-based merging on the CLIP ViT-B/32 eight-task suite (85.9% versus 69.1% for Task Arithmetic). This shows that selection quality matters. However, it assumes identical coordinates, and it still scores parameters once. We will include its connectivity score as an alternative one-shot selector inside our pipeline. If pursuit started from that score adds nothing, the pursuit claim fails.
Why not just distill the donor into the target? Distillation is the standard way to transfer across architectures, but it needs far more forward passes than 2,000 calibration samples, as Transport and Merge argues. We will run distillation at the matched token budget of the fine-tuning regime. If it wins there, weight transplantation has no advantage at this data scale, and we will say so.
Why not just route between the models? A mixture-of-experts wrapper such as MixtureKit avoids merging entirely, but it keeps every model in memory. FlexMerge shows that this linear cost is exactly what merging exists to avoid. We therefore treat routing as an upper reference, not a competitor.
Proposed Experiments
The first experiment is a synthetic check. We plant a known set of donor neurons in a random MLP and vary activation correlation. We then verify that pursuit recovers the planted set where top-$k$ fails, and that the crossover tracks the measured restricted condition number.
The second experiment is the vision testbed. We use CIFAR-100 and ImageNet with width-mismatched ResNets, with the HeteroZ-LMA baseline and our selection swapped in for zipping. We report joint accuracy against the fraction of neurons transplanted.
The third experiment is the main LLM study. We use the Transport and Merge donor/target pairs and evaluate on MalayMMLU, CMMLU and MGSM Thai, in both the training-free and matched-fine-tuning regimes.
The decisive check has two parts. First, across all layers and model pairs, the per-layer improvement of pursuit over top-$k$ with refit should correlate positively with the measured restricted condition number. Second, the training-free variant should close most of the gap to fine-tuned Transport and Merge. If the correlation is absent, the theory adds nothing beyond "refit after selecting," and the proposal reduces to that simpler, still useful result.
References
- Transport and Merge: Cross-Architecture Merging for Large Language Models
- Exact Recovery of Hard Thresholding Pursuit
- A Tight Bound of Hard Thresholding
- Orthogonal Matching Pursuit with Replacement
- Training-free Heterogeneous Model Merging
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Saliency-Aware Model Merging
- MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models
- Navigating the Accuracy-Size Trade-Off with Flexible Model Merging