When a hospital has fifty patients with a rare cancer, it cannot fine-tune a large model on them. What it can often do is borrow. Models already trained on larger related cohorts can be combined, and a model's "task vector" (the difference between its fine-tuned and pretrained weights) is a convenient unit to borrow. The open question is how to decide, from fifty patients, which source should supply which part of the network. We propose to learn one mixing coefficient per source for each block of parameters (a layer, an attention head, an output row) and to regularize those coefficients with the exclusive lasso. The exclusive lasso is a penalty that makes the members of a group compete, so that inside each group only a few survive. Here each parameter block is a group, and the sources compete to own it. Every block still gets filled, because the penalty spreads capacity across blocks, but each block is mostly taken from one or two sources rather than from a blurred average of all of them.
The second ingredient answers the question every small-sample regularizer faces: how do you set the penalty strength when there is too little data to cross-validate? We add "decoy" task vectors, obtained by fine-tuning on source data with shuffled labels, to the pool of competitors. We then choose the smallest penalty at which the decoys lose every block. This reuses the artificial-feature trick from Correlated Feature Selection with Extended Exclusive Group Lasso, but moves it from input features to model parameters. The resulting claim can be tested directly. With a small fixed number of learned coefficients, calibrated exclusive competition among sources should transfer better at $n \le 100$ than uniform merging, source-level selection, or the heuristic top-$K$ expert cut used in current whole-slide prognosis transfer. It should also refuse sources that carry no label signal.
Where this sits
Exclusive sparsity has a record in exactly the regime we care about, but only in linear models. Exclusive Lasso for Multi-task Feature Selection groups one feature across competing tasks, applies a squared $\ell_1$ penalty within the group, and beats both $\ell_1$ SVMs and the cooperative group lasso with only 10 samples per category. Our construction transposes that paper: its tasks competing for a feature become our sources competing for a parameter block. Learning a Product of Experts with Elitist Lasso shows that forcing ensemble experts to compete for features produces a division of labor that helps when the dominant signal is missing. That result motivates our expectation that exclusivity should matter most when sources are redundant. Combined Group and Exclusive Sparsity for Deep Neural Networks (CGES) brings exclusive sparsity into network weights, but it trains everything from scratch on CIFAR and ImageNet. We never touch the weights themselves, only a few thousand merge coefficients. That keeps the non-smooth proximal step cheap, which addresses the overhead concern the file raises about deep exclusive sparsity. On the tuning side, Correlated Feature Selection with Extended Exclusive Group Lasso supplies the artificial-competitor idea, and it also constrains our design: it needed hundreds of subsample refits for stability selection. We budget for bootstrap stability only as a diagnostic, not as the selection mechanism.
On the transfer side, Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis (STEPH) is the closest existing work and our main competitor. It mixes task vectors from TCGA cancer types through hypernetworks and keeps the top-$K$ ($K=5$) experts per input. Its sparsity is a hard, input-conditioned cut at the level of whole sources. It never penalizes two sources for supplying the same direction, and its hypernetworks are themselves learned parameters that must be fit from target data. We differ in three ways: competition happens per parameter block rather than per source, the penalty explicitly punishes co-ownership, and there are no hypernetwork parameters to fit. Boosting Transfer Learning with Survival Data from Heterogeneous Domains (Transfer Survival Boosting, TSB) handles negative transfer by re-weighting source patients by similarity to the target. That is instance-level borrowing, whereas ours is parameter-level borrowing, and we treat TSB as the reference method on its own benchmark. Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data motivates the decoy test: naive transfer into small cohorts imports dataset artifacts. A borrowing method should therefore be able to show that it rejects sources carrying no label signal.
Core Claim
Write $\theta_0$ for the shared pretrained weights, $\tau_s$ for the task vector of source $s \in \{1,\dots,S\}$, $\tau_{s,b}$ for its restriction to parameter block $b \in \{1,\dots,B\}$, and $\alpha_{s,b}$ for a learned scalar. The merged target model is
$$\theta(\alpha)_b = \theta_{0,b} + \sum_{s=1}^{S} \alpha_{s,b}\, \tau_{s,b},$$
and $\alpha$ is fit on the $n$ target patients by minimizing
$$\mathcal{L}_{\text{Cox}}(\theta(\alpha)) + \lambda \sum_{b=1}^{B} \Big( \sum_{s=1}^{S} |\alpha_{s,b}| \Big)^2 ,$$
where $\mathcal{L}_{\text{Cox}}$ is the Cox partial likelihood on target survival data and $\lambda$ sets the strength of competition. The squared inner $\ell_1$ term makes sources compete within a block. The outer sum acts like ridge across blocks, so no block is left empty. We solve with proximal gradient, using the closed-form exclusive-lasso proximal step, which needs one sort per block (A highly efficient algorithm for solving exclusive lasso problems).
To calibrate $\lambda$, we add $D$ decoy vectors $\tilde\tau_d$ to the pool. Each is fine-tuned from $\theta_0$ on a source cohort with survival labels permuted, so it carries the drift of fine-tuning but no label signal. We set
$$\lambda^\star = \min\{\lambda : \alpha_{d,b}(\lambda) = 0 \ \text{for all decoys } d \text{ and blocks } b\},$$
and then refit without decoys at $\lambda^\star$. This uses only training data and no held-out split. With $n = 50$, that matters: nested cross-validation would leave roughly ten patients per fold.
Research Program
The main testbed is the TCGA 13-cancer prognosis collection used by STEPH, with frozen UNI patch features. Each patient is represented by a bag of patch features, and an attention-pooled survival model built on them serves as the shared architecture. We pretrain $\theta_0$ on a pan-cancer pool that excludes the target cancer. From that shared initialization we fine-tune one task vector per remaining cancer type, plus three decoys. Each cancer in turn becomes the target, with its training set subsampled to $n \in \{25, 50, 100\}$ patients over 20 random draws and evaluated on a fixed held-out test set. Rare types such as sarcoma, which STEPH highlights as the hard case, are reported separately.
We compare three block definitions at increasing granularity: per layer, per attention head or neuron group, and per output row. The goal is to find where competition stops helping and starts overfitting. The number of coefficients is $S \times B$, which ranges from about a hundred to a few thousand.
The second testbed is the MAGGIC heart-failure studies used by TSB. We train a small Cox multilayer perceptron per source study, using shared features after harmonization, and target DIAMO, DIG, ECHOS, Euro and IN-CH one at a time. This checks that the effect is not specific to pathology features or to attention models.
The analyses go beyond accuracy. We record ownership maps, meaning which source wins which block and whether that pattern is stable across the 20 subsamples. We record decoy leakage, meaning the total weight decoys receive under each method. We also run a controlled redundancy test, in which we duplicate a source using a second fine-tuning seed. Exclusive competition should give each block to only one of the two copies, while ridge-style and softmax mixing should split the block between them.
Baselines
The naive reference is a target-only model. It fine-tunes the survival model from $\theta_0$ on the $n$ target patients with weight decay, using the same optimizer and number of epochs as every other method. A second naive point is $\theta_0$ with only a retrained linear risk head. Uniform task arithmetic, which adds all task vectors with one shared coefficient, is the simplest borrowing reference.
The strongest existing method is STEPH (Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis). We run it on the same task vectors, with $K=5$ as published, and its hypernetworks are trained on the same target samples. The file reports STEPH's gains only as relative C-index improvements: an average of 5.14% over single-cancer models and 2.01% over earlier transfer baselines on the 13 TCGA cohorts. No absolute per-cancer C-index was found on Lacuna for STEPH, and no reported result was found for any method at our subsampled $n \le 100$ targets. We will therefore reproduce STEPH under our splits and report both its full-data and subsampled numbers. On MAGGIC, the reference is TSB (Boosting Transfer Learning with Survival Data from Heterogeneous Domains), which reports a C-index of 0.677 on the Euro study against 0.500 for a target-only Cox model. We run TSB on identical target subsets.
Three ablations isolate the key ingredient by holding the per-block parameterization fixed and changing only the penalty. The first uses per-block coefficients with an $\ell_2$ penalty, which gives no competition. The second uses per-block coefficients with a plain $\ell_1$ penalty, which gives sparsity without the squared-sum competition, so whole blocks can go empty. The third uses a per-block softmax over sources with an entropy penalty, which is the most dangerous alternative. A group-lasso variant that switches whole sources on or off tests cooperative against competitive grouping. A last pair of ablations separates the penalty from its calibration: exclusive penalty tuned by nested cross-validation versus by decoys, and decoys added to the $\ell_2$ variant with no exclusive penalty.
Every method sees the same frozen features, task vectors, splits, epoch budget and a matched tuning budget of at most 20 configurations. Each method may use nested cross-validation, except the proposal's decoy variant, which uses none. Beating STEPH means a higher mean C-index across targets at $n \le 100$, with a paired Wilcoxon test over the 20 draws times 13 targets, plus lower decoy leakage. A tie at full data is acceptable if the small-$n$ gap holds.
Why not just…?
Why not just fine-tune on the target with strong weight decay or a low-rank adapter? At $n=50$, a few thousand merge coefficients are far fewer free parameters than even a rank-4 adapter on an attention model. The COVID study found that small-cohort training learns scanner and dataset artifacts rather than pathology (Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data). The target-only baselines settle this directly. If they match merging at $n=100$, we report that as the point where borrowing stops paying.
Why not just select whole sources, as STEPH's top-K or a group lasso does? Source-level selection assumes a source cancer is useful everywhere or nowhere. Our redundancy test and ownership maps check whether different blocks prefer different sources. If ownership maps collapse to a few sources owning every block, the per-block machinery is unnecessary and group lasso or STEPH should tie. We would then report the method as equivalent in accuracy, with the decoy calibration as its remaining contribution.
Why not just put a per-block softmax with an entropy penalty on the coefficients? This is a real threat. It also produces one or two winners per block and is smooth. The differences are that exclusive lasso lets a block's total borrowed magnitude shrink toward pure $\theta_0$, and that it yields exact zeros, which makes decoy calibration a crisp rule rather than a threshold. If softmax plus entropy matches our C-index and decoy leakage, the penalty adds nothing. The claim would then reduce to decoy-based calibration, which we can apply to the softmax variant, and we would reframe the paper that way.
Why not just tune λ by cross-validation? At $n=25$, each fold holds about five patients, so C-index estimates on a fold are close to noise. The ablation that compares nested cross-validation with decoy calibration measures the variance of the selected $\lambda$ and the resulting test C-index across draws.
Decisive Checks
Three results would confirm the proposal. First, the exclusive penalty with decoy calibration should beat both STEPH and the softmax-with-entropy ablation in mean C-index at $n \in \{25, 50\}$ on the TCGA targets. Second, it should assign near-zero weight to decoys while the $\ell_2$ and softmax variants do not. Third, in the duplicated-source test, it should give each block to one copy in most blocks. On MAGGIC, it should at least match TSB's reported 0.677 on Euro under full target data and degrade more slowly as $n$ shrinks.
Two results would count against it, and we would publish either. One is decoys receiving weight under $\lambda^\star$ calibration. The other is the advantage over plain per-block $\ell_1$ vanishing, which would mean the squared-sum competition is not doing the work.
References
- Correlated Feature Selection with Extended Exclusive Group Lasso
- Exclusive Lasso for Multi-task Feature Selection
- Learning a Product of Experts with Elitist Lasso
- Combined Group and Exclusive Sparsity for Deep Neural Networks
- Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis
- Boosting Transfer Learning with Survival Data from Heterogeneous Domains
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- A Highly Efficient Algorithm for Solving Exclusive Lasso Problems