When a medical imaging model has only a few hundred labeled examples for the task that matters, practitioners add auxiliary tasks. These can be self-supervised reconstruction on unlabeled scans, or related labeled datasets. Auxiliary tasks sometimes help and sometimes quietly hurt, and current fixes give no view of where the harm happens. The best of them, ForkMerge, trains a target-only copy and a target-plus-auxiliary copy of the network, then blends all of their weights with one global coefficient. That coefficient is chosen on a held-out labeled validation set. In the scarce-label regime, that validation set is exactly what we lack.
We propose to replace the single global blend with a per-neuron one. Interpretability tools now make it cheap to measure how much each hidden unit contributes to a prediction, and those measurements need no labels. We use them to find the small set of neurons the target task relies on and to shield those neurons from auxiliary updates. Every other neuron is left free to absorb whatever the auxiliary tasks teach. The resulting merge rule needs no labeled validation data. As a by-product it produces a map of which parts of the network the tasks share and which parts they fight over.
Core Claim
In auxiliary-task learning with very few target labels, we claim that most harmful auxiliary updates are concentrated in a small, identifiable set of neurons that carry the target prediction. We test this on the inner units of each Transformer feed-forward block. If so, restricting the merge to those neurons should do two things. It should match or beat a globally tuned blend at 1% labels. It should also have far lower variance across random choices of the tiny validation split, because the protected neurons are chosen from unlabeled images rather than from labels.
The claim is falsifiable in two ways. The relevance-guided mask might do no better than a random mask of the same size. Or it might do no better than a mask built from simpler per-parameter importance scores. In either case the interpretability ingredient is doing no work.
Why This Is Not Already Done
Three lines of work each touch part of the question.
The first is multi-task optimization. ForkMerge argues that negative transfer comes from a shift in the target distribution induced by auxiliary data, not from gradient conflict itself. It filters harm by linearly interpolating the two branches' full parameter vectors with a validation-chosen $\lambda^*$. RotoGrad rotates the shared feature space per task and rescales gradient magnitudes. Both treat the network as an undifferentiated vector, and both were evaluated with full supervision. Neither says which units are harmed. ForkMerge's dependence on a representative validation set is its acknowledged weak point, and it motivates this proposal directly.
The second is circuit discovery, which supplies the method. Language Model Circuits Are Sparse in the Neuron Basis shows three things. The activations inside the MLP, taken before its down-projection, form a "privileged basis," meaning individual neurons are meaningful units. Relevance Patching (RelP), a one-backward-pass attribution, assigns each neuron a share of the output. That attribution satisfies completeness, meaning the shares sum to the output. Faithful circuits of about 200 neurons exist for linguistic tasks. Discovering Influential Neuron Path in Vision Transformers gives the vision-side evidence we need: keeping only the top 5 feed-forward neurons per layer of ViT-B-16 preserved 67.7% ImageNet accuracy, where the pruning baseline ViT-Slim fell to about 10%. ACDC and S³martCirc automate circuit discovery, but only for single behaviors in language models. They rely on many clean and corrupted prompt pairs. None of these papers uses a circuit to steer training.
The third is using localization for an engineering goal. Task-Circuit Quantization keeps task-salient weights at full precision during compression. Neural Knowledge Bank adds feed-forward memory slots so that new knowledge avoids overwriting old knowledge; standard fine-tuning dropped summarization from 30.1 to 24.9 ROUGE-L. Both show that localized protection beats global updates. Neither addresses multi-task interference or label scarcity.
Our contribution is a merge rule for multi-task learning. It targets the label-scarce regime, is guided by attribution, and is validation-free. We also give the first measurement of whether negative transfer is localized at the neuron level.
The application regime comes from label-efficient radiology. Masked Record Modeling (MRM) pretrains a ViT on MIMIC-CXR by reconstructing masked image patches and masked report tokens. It reports 88.5% mean AUC on CheXpert with 1% of labels, and 73.2 Dice on SIIM pneumothorax segmentation with 10%. MRM gives us a strong starting encoder. It also shows that auxiliary objectives matter here, since removing report modeling cost about 5%. But MRM never asks whether its objectives interfere, or where.
Research Program
We follow ForkMerge's schedule. Every $\Delta t$ steps, training forks into branch 0, trained on the target loss alone, and branch 1, trained on target plus auxiliary losses. The change is in how the branches are merged.
At each merge point we run RelP on branch 0. We use unlabeled target-domain images and attribute the target head's own predicted logits, so no labels are consumed. This gives each feed-forward neuron $i$ a relevance $r_i \ge 0$. Because of completeness, the relevances sum to the attributed output, so "fraction of the target's relevance" is a meaningful quantity.
Let $P_\rho$ be the smallest set of neurons whose relevance sums to a fraction $\rho$ of the total. We fix $\rho = 0.8$ in advance, echoing the roughly 80% faithfulness level at which RelP circuits were reported. Each neuron owns one row of the up-projection and one column of the down-projection; call these parameters $\theta_i$. We merge them as
$$\theta_i^{\text{new}} = \theta_i^{(0)} + m_i\,\big(\theta_i^{(1)} - \theta_i^{(0)}\big), \qquad m_i = \begin{cases} 0 & i \in P_\rho \\ 1 & i \notin P_\rho \end{cases}$$
Here $\theta^{(0)}$ and $\theta^{(1)}$ are the two branches' parameters. Attention and normalization parameters follow branch 1 by default. A variant blends them with a ForkMerge-style scalar, which isolates how much of any gain comes from the neuron mask.
Three things are built and analyzed. The first is the merge rule itself, together with a softer variant in which $m_i$ decays with $r_i$. The second is a precondition check that the privileged-neuron picture holds in a radiology ViT at all. This is the open question the file raises between SAE-Rad, which argues these features require sparse autoencoders, and the neuron-basis paper, which argues they do not. The third is a diagnostic study of the question behind the method: when an auxiliary task hurts, do the neurons whose target relevance drops between branches cluster in a few layers and a few percent of units, or is the damage diffuse? If the damage is diffuse, the method has no reason to work, and we will report that as a negative finding about the structure of negative transfer.
Baselines
Naive references. Two naive reference points anchor everything. The first fine-tunes the MRM-pretrained encoder on the target labels alone. The second trains with a plain summed loss over target and auxiliary tasks. The first is what a practitioner does without auxiliary tasks. The second is the naive multi-task baseline, which ForkMerge reports can lose 10–15% against single-task training on DomainNet. The reported target number for the pretrained starting point is MRM's 88.5% mean AUC on CheXpert at 1% labels, from MRM. We will reproduce it in our pipeline before comparing anything else.
Strongest existing method. The strongest competitor is ForkMerge itself, with $\lambda^*$ chosen by line search on a validation split carved from the same 1% label pool. This is the method our proposal most directly modifies, and it already beats gradient-surgery methods such as NashMTL and CAGrad. Its reported results are a 4.03% relative improvement over single-task learning on NYUv2 and +2.00% on DomainNet. Both were obtained with full supervision; we found no reported result on Lacuna for ForkMerge under 1% or 10% labels. We also run RotoGrad as a second optimization-based competitor.
Fairness conditions. All methods share the encoder, the auxiliary data, the fork interval, the number of optimizer steps and a matched hyperparameter search budget. We count the extra RelP backward passes against our method's compute.
What counts as beating ForkMerge. Beating it means equal or higher test AUC on the 500-study radiologist-consensus CheXpert test set from CheXpert, together with a clearly smaller spread across 20 resampled validation splits.
Ablations. Three ablations isolate the key ingredient. The first replaces RelP relevance with a random neuron set of equal size. The second uses per-parameter Fisher or weight-magnitude importance. The third protects whole layers instead of individual neurons, chosen by summed relevance.
Why not just…?
Why not just tune a per-layer $\lambda$ in ForkMerge? Per-layer coefficients add structure, but they multiply the number of quantities estimated from a tiny validation set. At 1% labels that should make variance worse, not better. The layer-level ablation, where masking is chosen by relevance with no labels, separates two questions. One is whether finer-than-global structure helps. The other is whether neuron granularity specifically helps. If per-layer tuning matches us at equal variance, the neuron-level claim fails.
Why not just use Fisher or magnitude importance instead of an interpretability method? This is the real threat. Fisher importance is also a gradient-based measure of how much a parameter matters. We have two reasons to expect RelP to do better. It attributes to neurons that the neuron-basis paper shows are aligned with coordinate axes, rather than to raw weights. And its completeness property makes the fixed $\rho$ threshold meaningful without validation. But this is an expectation, not evidence from the file. The Fisher ablation, run at identical protected-set sizes, settles it. If Fisher ties, we will report that the interpretability tool adds interpretive value but no accuracy.
Why not just freeze the pretrained encoder and train a linear probe? Freezing avoids negative transfer entirely, and MRM is already strong at 1% labels. But it also forfeits any auxiliary benefit, and MRM's own ablation shows auxiliary objectives are worth several points. The single-task and frozen-probe references tell us whether auxiliary learning helps at all in this setting. If neither ForkMerge nor our method beats them, the question is moot, and we will say so.
Proposed Experiments
The primary setting uses an MRM-style ViT pretrained on MIMIC-CXR, reproduced if released weights are unavailable. The target is CheXpert classification at 1% and 10% labels. We test three auxiliary sources: continued masked-image reconstruction on unlabeled MIMIC-CXR, NIH ChestX-ray classification, and RSNA pneumonia detection. A second target is SIIM pneumothorax segmentation at 10% labels, scored by Dice, with the classification tasks as auxiliaries. To show the effect is not radiology-specific, we repeat the main comparison on NYUv2 and DomainNet with labels subsampled to 1% and 10%. These are ForkMerge's own benchmarks, so its full-label numbers serve as a sanity anchor. Metrics are mean AUC, Dice, a negative-transfer rate (the fraction of target–auxiliary pairings that fall below single-task training), and the standard deviation across validation resamples and seeds.
Precondition check. Before any merging, we run RelP on the radiology ViT and measure faithfulness against circuit size. If keeping about 80% of relevance does not require a small minority of neurons, the approach is abandoned.
Localization check. For auxiliary tasks that ForkMerge's line search rejects ($\lambda^* \to 0$), we test whether the neurons with the largest relevance drop between branches concentrate in few units. We then test whether reverting only those units recovers most of the target loss.
Head-to-head comparison. Our validation-free merge is compared against ForkMerge at 1% labels. Success means similar mean AUC with lower variance.
Ingredient check. The RelP mask must beat the random and Fisher masks of equal size. If it does not, the paper's contribution reduces to the diagnostic finding about where negative transfer lives, which is still worth reporting.
References
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Language Model Circuits Are Sparse in the Neuron Basis
- Discovering Influential Neuron Path in Vision Transformers
- Towards Automated Circuit Discovery for Mechanistic Interpretability
- S^3martCirc: Self-supervised Smart Circuit Discovery
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- Neural Knowledge Bank for Pretrained Transformers
- Advancing Radiograph Representation Learning with Masked Record Modeling
- An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison