Compact sequence models pretrained without labels are unusually good at medical time series. On electrocardiograms (ECGs), a small state-space model is one example. A state-space model is a recurrent network whose hidden state evolves step by step through a learned transition, and this one was trained by predicting its own future. It beats Transformer models 25 times its size, and pretrained ECG models match from-scratch training with only 10–30% of the labels. Deployment targets such as wearables, bedside monitors, and implants need these models smaller still. The pruning methods built for recurrent and state-space networks were all developed and scored on fully labeled or large-corpus tasks: language modeling, Sequential MNIST, zero-shot language benchmarks. Nobody has asked whether a pruned model keeps the thing that made it worth pretraining, namely its ability to learn a new diagnosis from a handful of labeled examples, especially for rare conditions.
We propose to decide which weights to remove using the model's own self-supervised objective, computed on unlabeled recordings from the target domain, rather than the scarce downstream labels or a generic calibration corpus. Pruning damage that disappears when there are plenty of labels for fine-tuning may be large when labels are scarce, and larger still for rare classes. We therefore evaluate every compression method by how much of the pretrained model's label-efficiency advantage it keeps, not by full-data accuracy.
Core Claim
Our claim has two parts. The first is a measurement claim: at moderate-to-high sparsity, differences between pruning criteria grow as the downstream label budget shrinks, and they are largest on rare classes. Full-label benchmarks therefore understate compression damage in exactly the regime these models are built for.
The second is a method claim. Take a state-space model pretrained with Contrastive Predictive Coding (CPC), a self-supervised method that trains the network to pick out its true future latent representation from distractors. Scoring weights by their contribution to the CPC loss on unlabeled target-domain ECGs keeps more of the low-label performance than three alternatives: label-gradient scoring, magnitude scoring, and layer-wise reconstruction scoring. It also does so better than pretraining a small dense model directly.
The mechanism is specific to sequences. One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation shows that loss gradients from a single end-of-sequence label fade over time. Gradient-based pruning then mistakes long-memory weights for dead ones, and SNIP reaches roughly 89% error on Sequential MNIST at 95% sparsity. A CPC loss predicts at many horizons, and every horizon-$k$ term sends a direct gradient through $k$ steps of the state transition. The pretext loss therefore probes memory at every timescale the model was trained to use, and it needs no labels at all.
Research Program
The backbone is a selective state-space model in the Mamba family, chosen so that the SSM-specific pruning of SparseSSM applies directly. It is pretrained with CPC on the PhysioNet 2021 and MIMIC-IV-ECG records that ECG-FM used for pretraining. We train it at roughly ten times the 3.8M-parameter scale of ECG-CPC from Benchmarking ECG FMs, so that compressing it to about ECG-CPC size is a meaningful target. We use 100 Hz input, which Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports loses little diagnostic performance relative to 500 Hz.
For each prunable weight $w_i$, the pretext saliency is the empirical Fisher score on unlabeled target-domain records $x$:
$$s_i = \mathbb{E}_x\Big[\big(w_i \,\partial \mathcal{L}_{\text{CPC}}(x)/\partial w_i\big)^2\Big], \qquad \mathcal{L}_{\text{CPC}} = \textstyle\sum_{k=1}^{K} \lambda_k \mathcal{L}_k .$$
Here $\mathcal{L}_k$ is the contrastive loss for predicting the latent $k$ steps ahead, $K$ is the longest horizon, and $\lambda_k$ is a per-horizon weight. We set $\lambda_k$ uniform by default. Uniform horizon weights deliberately replace the power-law emphasis on early tokens that SparseSSM uses. In a 10-second ECG, arrhythmic beats can occur anywhere in the window, not only in the prefix.
We apply the score in two forms. The unstructured form is a mask over all weights. The structured form removes whole columns of the state matrix, which SparseSSM reports gives 1.5–2.2× wall-clock speedups. After pruning comes a short "pretext healing" phase of sparse CPC training on unlabeled data only, with the mask fixed. We then fine-tune on the labeled subset. This pipeline never spends a label on the compression decision, which matters when there may be only a few hundred labeled records.
The analysis is organized around a label-efficiency retention curve. For each method and each sparsity level, we measure how many labels the pruned model needs to match a from-scratch supervised model trained on all labels. We compare that number with the 3.3–9× label-efficiency multiplier the ECG benchmark reports for dense models. We report the curve separately for common and rare PTB-XL statements.
Baselines
The naive reference points are the dense pretrained model and a supervised S4 trained from scratch at the compressed size. Benchmarking ECG FMs found that supervised S4 often beats much larger pretrained models, so it is not a strawman. For full-data context on PTB-XL, Self-supervised representation learning from 12-lead ECG data reports macro AUC 0.9418 for CPC fine-tuning, 0.9318 for the best supervised model, and 0.927 for a frozen linear probe, all with a dense xresnet1d. We found no reported result on Lacuna for any pruned ECG model, or for any model at 1–10% PTB-XL labels. Those numbers must be produced here.
The strongest existing compression method is SparseSSM: one-shot second-order pruning of the state-space and feed-forward blocks that keeps Mamba-370M near its dense zero-shot accuracy of about 47.5% at 50% sparsity, where magnitude pruning and SparseGPT collapse. It is the strongest competitor because it is already label-free: it reconstructs layer outputs on calibration data. The real question is therefore whether a global pretext objective beats a local reconstruction objective, not whether any label-free method beats label-based pruning. We run it twice, once with its original power-law temporal weighting and once with uniform weighting. Both runs use the same unlabeled target-domain calibration records, the same sparsity levels, and the same healing and fine-tuning budget as our method. It is beaten if our method keeps a larger label-efficiency multiplier, or gives higher rare-class macro AUC at the 1% and 5% label budgets across patient-split folds, at matched sparsity.
The other compression baselines are the following:
- Magnitude pruning of the pretrained model.
- Label-gradient saliency computed on the available labeled subset, the natural "use what labels you have" choice.
- Jacobian-spectrum pruning at initialization, followed by CPC pretraining of the sparse network. This is the method of One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation.
- Dynamic sparse CPC pretraining with random regrowth, as in Selfish Sparse RNN Training, at equal training compute.
Every method gets the same hyperparameter search budget per label fraction, selected on patient-disjoint validation folds.
Two ablations isolate the key ingredients. The first swaps the CPC loss for layer-wise reconstruction while keeping everything else fixed; this tests the objective. The second removes pretext healing; this tests whether the gain comes from choosing the mask well or merely from extra unlabeled training.
Why not just…?
Why not just pretrain a small dense model with CPC and skip pruning? This is the most serious threat. ECG-CPC is already small, and a dense model of the target size gets the same unlabeled data. We make it the primary comparison: a dense CPC model with parameters matched to each sparsity level, given pretraining compute equal to large-model pretraining plus pruning plus healing. If it matches pruned models on the retention curve, the honest conclusion is that compression is unnecessary for label efficiency at this scale. The remaining case for pruning would then rest on structured-pruning latency, which we measure separately.
Why not just prune by magnitude and let fine-tuning recover? At full labels this probably works, and our measurement claim predicts exactly that. SparseSSM already reports magnitude pruning collapsing on Mamba above 30% sparsity. Our experiment extends the test to the case where recovery can only use a few labels. If magnitude pruning with pretext healing closes the gap, the criterion does not matter, and we will say so.
Why not just distill into a small student on unlabeled ECGs? The usual objection is that distillation needs a transfer set. That objection is weak here, because unlabeled ECGs are plentiful. We run feature distillation from the large CPC teacher into a dense student of target size on the pretraining corpus, at matched compute. Benchmarking ECG FMs finds that different architectures learn very different representations. That finding argues for a same-architecture student, not against distillation. The retention curve settles whether distillation matches pretext pruning.
Why not just freeze the backbone and train a linear probe? Freezing does nothing for model size. Also, CPC fine-tuning beats the frozen probe on PTB-XL (0.9418 vs 0.927), and frozen features lag on structural and outcome targets in the ECG benchmark. We still report probe results for every pruned model, since probes are the most label-frugal use.
Proposed Experiments
The main benchmark is PTB-XL (PTB-XL): 21,837 ten-second 12-lead records with 71 statements, evaluated under the official patient-split folds. We use label fractions of 1%, 5%, 10%, 30%, and 100%, and sparsities of 50%, 70%, and 90%, both unstructured and state-column structured. Metrics are macro AUC overall and on statements below a fixed prevalence threshold, the label-efficiency multiplier, and CPU wall-clock latency and peak memory. To test transfer beyond PTB-XL, we repeat the 10% condition on ICBEB2018 and on the benchmark's cardiac-structure and acute-care categories. Those are the categories where ECG-CPC was strongest and where dense models show a gap on hidden physiological targets.
Three checks decide the question. First, does the spread between pruning criteria at 90% sparsity grow monotonically as labels shrink? If not, the measurement claim fails. Second, does pretext saliency beat SparseSSM with uniform weighting, and not just the power-law version? If only the latter, the gain comes from temporal weighting rather than the objective, which is a narrower finding. Third, does any pruned model beat the parameter-matched dense CPC model at the 1% and 5% budgets? If not, we recommend small dense pretraining. A positive outcome yields a label-free recipe that shrinks ECG models to wearable scale while keeping their few-label advantage on rare diagnoses.
References
- One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
- SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
- ECG-FM: an open electrocardiogram foundation model
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Self-supervised representation learning from 12-lead ECG data
- Selfish Sparse RNN Training
- PTB-XL, a large publicly available electrocardiography dataset