Sparse From Birth: Data-Free Pruning Before Self-Supervised Pre-training

Medical models are usually built by pre-training a large network on unlabeled scans or signals and then adapting it with a small number of expert labels. Both stages run on a fully dense network, where every possible connection between neurons is present. A separate line of work shows that you can choose a sparse wiring pattern for a network before training, using no data at all, and still get a network that trains well even when 99% of its connections are removed. This proposal asks a simple question: what happens if you cut the network down with one of these data-free methods first, then do self-supervised pre-training on unlabeled clinical data, then fine-tune on 1% or 10% of the labels? Two outcomes would matter. Pre-training compute and model size would shrink by an order of magnitude. And the sparse wiring might act as a built-in regularizer, a constraint that keeps the model from fitting noise in the few labeled examples, at exactly the stage where dense models overfit.

The second outcome has some support. Sparse Winning Tickets are Data-Efficient Image Recognizers found that sparse subnetworks beat dense models by up to 16% absolute accuracy on 1% subsets of CIFAR-10, and also beat dense models with the same parameter count. Those masks, however, came from iterative magnitude pruning, which needs repeated supervised training runs. That paper also never involved self-supervised pre-training. No study in the file connects data-free pruning at initialization to self-supervised pre-training or to low-label clinical benchmarks. We also make the claim falsifiable on one specific point: whether the wiring pattern matters, or only how many connections each layer keeps.

Core Claim

The claim has two parts. First, at high sparsity (90–98%), an encoder pruned at initialization with a criterion that balances paths and nodes, then pre-trained with self-supervision, matches or beats two alternatives at the same sparsity on 1%-label fine-tuning: a dense encoder pre-trained the same way and then pruned afterward, and a smaller dense encoder with the same parameter count. It should do this while using a fraction of the pre-training compute. Second, the advantage of a topology-aware mask over a mask that only preserves per-layer density should grow as labels get scarcer. If this holds, data-free topology is a regularizer for low-label learning and not merely a compression trick.

The second part is the one most at risk. Towards Data-Agnostic Pruning At Initialization: What Makes a Good Sparse Mask? reported a "mask-shuffling paradox": randomly permuting a mask's surviving connections within each layer often leaves accuracy unchanged. That result suggests per-layer density may explain most of the effect. Our experiments include the control that tests this directly.

Background and Positioning

The pruning methods we reuse form one line of increasingly refined data-free criteria. SynFlow pushes an all-ones input through the absolute values of the weights and prunes iteratively to preserve that "synaptic flow." This avoids layer collapse, where an entire layer is disconnected, which affects single-shot methods such as SNIP. SynFlow stays trainable up to 99.99% sparsity. PHEW builds masks from random walks that favor paths with large weight products.

Node-Path Balancing (NPB) explains why these masks work. Good masks keep many complete input-to-output paths and also many neurons that lie on at least one such path. SynFlow keeps paths but leaves few active neurons, producing narrow, stringy graphs. Random pruning keeps neurons but severs paths. DPaI turns the NPB objective into differentiable path, node and kernel terms and scales the approach to ViT-B/16. At 99% sparsity it reaches 35.61% accuracy, against 29.55% for SynFlow. NASI supports the premise that structure at initialization is largely independent of labels and data: its trainability proxy ranks architectures almost identically under true and random labels. These papers supply our method. All of them evaluate masks only under fully supervised training from scratch.

Two papers motivate and constrain the proposal. Sparse Winning Tickets, discussed above, motivates it: sparsity helps when data is scarce. The Emergence of Essential Sparsity in Large Pre-trained Models constrains it. That paper shows that after large-scale pre-training, one-shot magnitude pruning removes 30–60% of the weights at no cost, and that self-supervised backbones such as DINO are especially easy to prune. Its implication is that pruning before training is unnecessary at moderate sparsity. We accept this. We expect post-hoc pruning to be competitive near 50% and position the proposal at sparsities above 90%, where that paper reports one-shot pruning collapsing. Post-hoc pruning also saves nothing during pre-training.

On the label-efficient side, the pre-training recipes come from Self-supervised representation learning from 12-lead ECG data and Masked Record Modeling (MRM). The ECG paper uses Contrastive Predictive Coding (CPC), in which the model predicts its own future latent features, on an xresnet1d backbone. MRM uses masked image-and-report reconstruction on a ViT.

Research Program

The pipeline has three stages. First, we compute a sparse mask on the randomly initialized encoder using a data-free criterion. Second, we run self-supervised pre-training with the mask held fixed, so pruned weights stay at zero. Third, we fine-tune on a labeled subset. Only the encoder is pruned. The pretext heads, meaning the CPC predictor and the reconstruction decoder, stay dense, so that sparsity limits the representation and not the pretext task itself.

We add one ingredient that previous work lacks: a label-free way to choose the mask's free parameters. DPaI's balance between node and path terms needs tuning per architecture, which is a known open problem. Choosing it with labeled validation data would undercut the low-label premise. We instead choose it by the self-supervised loss on a held-out unlabeled split after a short pre-training run of about 5% of the full schedule. A secondary question is whether this unlabeled signal ranks masks the same way downstream 1%-label AUC (area under the ROC curve) does. If it does, mask design needs no labels at any point.

We also test whether the path and node metrics carry over to 1D temporal convolutional networks, an open question since NPB was defined and tested on 2D vision models. We measure effective paths and effective nodes for each mask on xresnet1d and regress downstream AUC on them.

The primary testbed is PTB-XL, chosen because it is cheap enough to run a full grid. PTB-XL contains 21,837 twelve-lead ECGs, described in PTB-XL, a large publicly available electrocardiography dataset. We use the CPC recipe on xresnet1d, pre-train on the unlabeled training records, and fine-tune with 1%, 10% and 100% of labels.

The secondary testbed is chest radiographs. We pre-train a ViT-S or ResNet-50 on MIMIC-CXR images with masked image modeling and evaluate on the five CheXpert competition pathologies (CheXpert) at 1% and 10% labels. We handle uncertain labels with one fixed policy across all arms, because CheXpert shows results are sensitive to that choice. We use SIIM pneumothorax Dice at 10% labels as a transfer check.

A third arm, run from scratch on 1% CIFAR-10 and ISIC, asks whether data-free masks reproduce the Sparse Winning Tickets result without iterative pruning. This links our result to that paper on its own benchmarks.

Unstructured sparsity reduces theoretical FLOPs but does not speed up dense GPU kernels. We therefore report FLOPs and parameter counts, and we report wall-clock time separately for the active channels identified by the node-preserving criteria, which can be physically removed.

Baselines

The do-nothing reference is the dense encoder with the same self-supervised pre-training and fine-tuning. For PTB-XL at full labels, the ECG paper reports a CPC fine-tuned macro AUC of 0.9418, against 0.9318 for purely supervised training and 0.927 for a linear probe. We must reproduce roughly these numbers before any comparison means anything. We found no reported result on Lacuna for PTB-XL at 1% labels, so we will establish the dense 1% number ourselves. For CheXpert at 1% labels, MRM reports 88.5% mean AUC with a ViT pre-trained on MIMIC-CXR images and reports. Our smaller image-only pre-training will likely fall short of this, so we treat it as an external ceiling rather than the bar to beat.

The strongest competitor at high sparsity is a dense encoder pre-trained with self-supervision, then pruned one-shot by magnitude to the target sparsity, then fine-tuned. This is the essential-sparsity recipe. It sees all the unlabeled data before choosing a mask, which is the main argument against pruning first. It gets the same pre-training steps (and therefore more FLOPs) and the same fine-tuning budget. We also include a variant with a short self-supervised retraining after pruning, so it is not handicapped by skipping recovery training. Beating it means higher 1%-label AUC at the same sparsity, with seeds giving non-overlapping 95% intervals, at 90% sparsity or above. At 50% we expect a tie.

The parameter-matched small dense model is a narrower xresnet1d or ViT-Ti with the same nonzero parameter count, pre-trained and fine-tuned identically. Sparse Winning Tickets found sparse beats size-matched dense from scratch. Whether this survives self-supervised pre-training is untested.

Two data-driven pruning baselines follow. SNIP, scored on a single unlabeled batch using the pretext loss, tests whether a little real data at initialization helps. Iterative magnitude pruning using only the small labeled subset reproduces the Sparse Winning Tickets procedure in the from-scratch arm.

The key ablations isolate the proposal's central ingredient. Random pruning with ERK, a heuristic that sets each layer's density from its shape, and a layer-shuffled DPaI mask both keep per-layer density while destroying the specific wiring pattern. If DPaI or NPB beat these, the topology itself carries the benefit. Every arm gets the same pre-training steps, the same fine-tuning sweep over learning rate and epochs (16 configurations, selected on the same small labeled validation split), and five seeds. We also report the sparse arms with pre-training extended to match the dense run's FLOPs.

Why not just…?

Why not just prune randomly with ERK densities? This is the most serious threat. The mask-shuffling paradox in NPB suggests the wiring within a layer may matter little once per-layer density is fixed. Against this, both NPB and DPaI show random masks lose many complete paths and degrade at high sparsity, and DPaI reports gains of up to 4.6% over prior pruning-at-initialization methods on ResNet at 99% sparsity. Those results are all for supervised training from scratch. Our shuffled-mask and ERK-random arms settle the question for self-supervised pre-training. If they match DPaI at 1% labels, we will report that per-layer density is what matters, and the topology-as-regularizer claim fails.

Why not just pre-train dense and prune afterward? Within 30–60% sparsity this probably works as well, per the essential-sparsity paper, which also reports one-shot pruning collapsing beyond that range. It also costs a full dense pre-training run. The sparsity sweep from 50% to 98% shows where the two curves cross, and we will report the crossover honestly.

Why not just train a smaller dense network? Sparse Winning Tickets found sparse tickets beat parameter-matched dense models in low-data training from scratch. Self-supervised pre-training could erase that gap, since the small dense model also gets a good representation. The matched-parameter arm tests this directly.

Why not just freeze the dense encoder and train a linear head? Freezing removes the overfitting problem without any pruning. The ECG paper's frozen CPC probe sits only 0.5% below supervised training at full labels. If a linear probe on the dense encoder matches our sparse fine-tuned encoder at 1% labels, the accuracy argument is gone and only the compute argument remains. We therefore report linear-probe results for every arm.

Why not just use SNIP on unlabeled data, since it is available? SynFlow shows single-shot gradient methods suffer layer collapse at 95–99% sparsity, and NPB reports gains of up to 14% over SNIP. The SNIP-with-pretext-loss arm tests whether self-supervised gradients change that picture.

Decisive Checks

The proposal succeeds if three things hold on PTB-XL at 1% labels. At 90–98% sparsity, the DPaI or NPB pre-trained encoder beats dense-then-prune and the parameter-matched dense model, with seeds giving non-overlapping intervals. It also beats the shuffled-mask control, and that margin is larger at 1% labels than at 100%. Finally, the mask chosen by unlabeled pretext loss falls within the top quartile of masks ranked by downstream AUC.

The chest radiograph testbed must reproduce the direction of these results. If the shuffled control ties DPaI, or if the dense linear probe matches the sparse encoder, the regularization claim fails. The work then reduces to a measured statement about how much pre-training compute data-free pruning saves at a given accuracy cost, which is still useful but smaller.