Recycling Dead Capacity Across CNNs, Transformers, and State-Space Models During Label-Scarce Sequential Adaptation

Medical foundation models are rarely adapted once. A hospital's ECG model is fine-tuned on a few hundred labels for arrhythmia, then on a few hundred more for a structural-heart target, then for a new site's hardware. Each round has a small label budget, and each round runs on the same weights. Work on loss of plasticity, meaning a network's gradual loss of the ability to learn new things after long training on shifting targets, shows that this kind of repeated adaptation can leave parts of a network permanently inactive. Current repair methods, however, are defined only for individual ReLU units in MLPs and CNNs. They do not cover the attention heads and state-space channels that make up today's clinical foundation models.

This note proposes one architecture-agnostic repair rule and one regime in which to test it. In any residual network, every block reads from a shared running representation (the residual stream) and writes back to it through an output projection. We define a "unit" as one read/write pair: an MLP hidden neuron, a convolutional channel, an attention head, or a state-space channel. We score each unit by how much it actually writes into the stream. Units that write almost nothing get fresh read-side weights and a zeroed write side, so the network's output does not change at the moment of repair. The claim is that this single rule detects and reverses capacity loss the same way across CNN, Transformer, and state-space ECG foundation models during sequences of 1%–10%-label adaptations. It should beat both naive sequential fine-tuning and the obvious fix of restarting from the pretrained checkpoint for every task, whenever earlier tasks contain knowledge that later ones can use.

Core Claim

The dormant-neuron work, The Dormant Neuron Phenomenon in Deep Reinforcement Learning, introduced ReDo. ReDo flags ReLU units whose average activation falls below a threshold, reinitializes their incoming weights, and zeroes their outgoing weights. Loss of plasticity in deep continual learning introduced continual backpropagation, which keeps reinitializing a small fraction of low-utility units. Both methods supply the mechanism we reuse, and both define "unit" as a scalar activation. That definition fails in three places. It fails for GeLU and SwiGLU activations, which are never exactly zero. It fails for attention heads, where the natural unit is a whole head. It fails for state-space layers, where a channel is a recurrence with input and output maps. Our proposal changes what counts as a unit and how it is scored. For block $\ell$ with unit $j$, let $h_j$ be the unit's read-side output (a neuron's activation, a channel's feature map, a head's attention output, or an SSM channel's state readout before the output map), and let $W^{\text{out}}_{:,j}$ be the slice of the block's output projection that carries it into the residual stream. The write utility is

$$u_{\ell j} = \frac{\mathbb{E}_x\big[\lVert h_j(x)\rVert\big]\,\lVert W^{\text{out}}_{:,j}\rVert}{\tfrac{1}{J_\ell}\sum_{k}\mathbb{E}_x\big[\lVert h_k(x)\rVert\big]\,\lVert W^{\text{out}}_{:,k}\rVert},$$

where $J_\ell$ is the number of units in the block and the expectation is over a buffer of recent inputs, which can include unlabeled ones. A unit with low $u_{\ell j}$ contributes nothing to the residual stream, whatever its activation function.

Repair has two parts. We reinitialize the unit's read-side parameters from the pretraining initialization distribution: the incoming weights for an MLP or convolution, the query, key, and value slices for a head, and the input and transition parameters for an SSM channel. We also zero $W^{\text{out}}_{:,j}$. Zeroing the write side means the repair causes no change in the model's predictions, which is the property that made ReDo safe in reinforcement learning. To avoid ReDo's hand-tuned threshold $\tau$, which the literature lists as a known weakness, we recycle a fixed small fraction of the lowest-utility units per block, as continual backpropagation does, and normalize scores within each block so that heterogeneous layers are treated comparably.

Why This Regime, and What It Is Not

Three lines of prior work bound the idea. Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning explains plasticity loss as collapse of the curvature spectrum. Its fix, L2-ER, combines weight decay with an SVD-based penalty that keeps feature rank high. That paper motivates our diagnostic, since we track feature rank, and it also constrains our design: it calls the extension to attention an open problem, and the SVD cost scales poorly. Our rule needs only running means of activation norms. Weight Decay Improves Language Model Plasticity shows that higher pretraining weight decay yields language models that fine-tune better. It acts on the pretraining side, on Transformers only, without label scarcity. We act during downstream adaptation, across three families. PLATE builds adapters from redundant neurons and a frozen input nullspace to trade plasticity against retention over 2–4 tasks. It assumes redundancy is already present; we create usable capacity when it runs out, over longer sequences.

There is an apparent tension with the anchoring school. Anchored Supervised Fine-Tuning adds a KL penalty that keeps the model close to its base outputs, and Proximal Supervised Fine-Tuning clips updates to a trust region. Both argue that drift destroys prior knowledge, which seems to contradict resetting. Zero-write recycling does not move the function at all at reset time, so it is compatible with an anchor in principle. We test the combination rather than assume it works.

The application regime comes from Benchmarking ECG FMs: A Reality Check Across Clinical Tasks. That benchmark compares CNN (ECGFounder), Transformer (HuBERT-ECG, ECG-JEPA), and state-space (ECG-CPC) foundation models on 26 tasks. It finds that the 3.8M-parameter SSM wins 5 of 7 categories, that all models reach 3.3x–9x label efficiency over training from scratch, and that the three families solve tasks with internal representations that look unrelated under CKA, a standard similarity measure. Its evaluations are all static. That is the reason this setting suits a test of architectural generalization: if one repair rule behaves consistently across three families whose internal representations differ this much, it relies on residual read/write structure and not on a particular latent geometry.

Research Program

The first stage is diagnostic and could produce a negative result. We build a task stream from the ECG benchmark's datasets, starting with PTB-XL (PTB-XL, a large publicly available electrocardiography dataset). PTB-XL's superclass, subclass, statement, and demographic targets come first, followed by structural and outcome targets from other benchmark datasets and then cross-dataset (cross-hardware) repeats. Each task gets a 1% or 10% label budget. One backbone per family is adapted sequentially, with task-specific heads and a shared backbone. Along the way we log the write-utility distribution, the fraction of units below 1% of the block mean, the feature effective rank, and a "plasticity gap." The plasticity gap is the AUC a backbone reaches on task $k$ after $k-1$ prior adaptations, minus the AUC a fresh copy of the pretrained checkpoint reaches on the same labels. If the gap stays near zero for every family over 20–30 tasks, plasticity loss is not a problem in this regime, and that finding alone is useful to the clinical-foundation-model community.

The second stage applies write-utility recycling with an identical recycle fraction and schedule in all three families. We tune it once per family on a held-out validation stream, under the same budget given to every baseline. The third stage checks that the rule is not ECG-specific. We run it on Continual ImageNet, a stream of 2,000 binary ImageNet tasks, with a small ViT and a small Mamba-style model next to the original CNN. Prior work there shows standard backpropagation dropping "below a linear baseline after 2,000 tasks" (Loss of plasticity in deep continual learning). As a third modality, a shorter CheXpert stream starting from an MRM-pretrained model (Advancing Radiograph Representation Learning with Masked Record Modeling) tests whether a vision-language Transformer that already reaches 88.5% mean AUC at 1% labels keeps that efficiency across repeated adaptations.

Baselines

The naive reference point is plain sequential full fine-tuning of each backbone through the stream with standard weight decay. It shows how much capacity is lost when nothing is done.

The strongest competitor is restarting from the pretrained checkpoint for every task and fine-tuning with LoRA or fully. It removes plasticity loss by construction, so it is the bar for raw plasticity. It gives up any transfer from earlier labeled tasks and requires keeping one model per task. It runs on identical label subsets and identical step counts. The file has no reported per-task AUC for this protocol on the ECG benchmark, so we generate it ourselves. For context, an ensemble of supervised CNNs reaches a macro AUC of about 0.939 on PTB-XL diagnostic statements with full labels (Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL), which sets a ceiling for low-label runs on that task.

The strongest existing plasticity methods are three. Continual backpropagation and ReDo are ported literally, applied only to scalar MLP or convolutional units with their original activation-based scores. L2-ER is run where its SVD is affordable. Shrink and Perturb (DrM) serves as the soft-reset baseline. No Continual ImageNet accuracy for continual backpropagation or L2-ER is reported in the file beyond qualitative "sustained" and "stable" descriptions, so we rerun them.

For retention, we run PLATE and ASFT-style anchoring. The relevant numbers are PLATE's 1.85% forgetting versus LoRA's 7–9% on vision (PLATE), and Block Expansion's 81.60% versus LoRA's ~77% versus full fine-tuning's 56.69% mean transfer accuracy on DINO ViTs (Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting). Every method receives the same hyperparameter search budget, the same unlabeled buffer, and the same compute.

Three ablations isolate the key ingredient. The first recycles the same fraction of randomly chosen units, which tests whether the utility score matters. The second scores units by read-side activation only, ReDo-style but at head and channel granularity, which tests whether the write-side norm matters. The third reinitializes without zeroing the write side, which tests whether zero distortion matters.

Beating the field means three results. Recycling must keep the plasticity gap within noise of the restart baseline in all three families. It must beat restart on later tasks through forward transfer. It must beat literal ReDo/continual backpropagation and L2-ER on the Transformer and SSM backbones, where they cover only part of the network.

Why not just…?

Why not just restart from the pretrained checkpoint for every task? This is the real threat. If the tasks share little, restarting will match or beat everything, and we will report that. The proposal's value depends on forward transfer: a backbone that has seen earlier ECG tasks should learn a 1%-label structural target better than a fresh one. The decisive comparison is late-stream AUC for recycling versus restart. Recycling must win while naive sequential fine-tuning loses. If restart wins, plasticity maintenance is unnecessary for clinical foundation-model adaptation at these stream lengths.

Why not just raise weight decay? Weight decay partly mitigates plasticity loss, but prior work found it insufficient. Backpropagation with $L_2$ still fell below linear baselines on Continual ImageNet (Loss of plasticity in deep continual learning), and ReDo outperformed weight-decayed DQN (The Dormant Neuron Phenomenon in Deep Reinforcement Learning). The language-model result (Weight Decay Improves Language Model Plasticity) concerns pretraining, not downstream streams. We include a tuned high-weight-decay arm in every family, and the comparison either settles the question or shows weight decay is enough.

Why not just freeze the backbone and train linear heads? A frozen backbone cannot lose plasticity, but the ECG benchmark reports that frozen features trail end-to-end fine-tuning on complex clinical categories (Benchmarking ECG FMs). Our probe-only arm measures how much of that gap remains at 1% labels, where probing should be most competitive.

Why not just apply ReDo or continual backpropagation to the MLP units that Transformers and SSMs already have? This is the literal port, and it may recover much of the benefit. If dormancy concentrates in attention heads or SSM channels, though, the MLP-only port will leave it untouched. The per-component utility logs from stage one show where dormancy concentrates, and the head-to-head comparison measures what the extra coverage is worth.

Decisive Checks

We consider three outcomes decisive. First, stage one must show that the plasticity gap grows over the stream for at least one family. The utility logs should show whether the SSM, given its recurrent channels, collapses differently from the Transformer. Second, with a single recycle fraction tuned per family, write-utility recycling must close that gap in all three families on both ECG and Continual ImageNet. Third, the random-unit and no-zeroing ablations must each perform measurably worse, which would show that the score and the zero-distortion write are doing the work. A further check follows from the anchoring tension. If recycling combined with an ASFT-style KL anchor retains earlier-task AUC as well as the anchor alone while also keeping plasticity, then resetting and anchoring are complementary rather than opposed.