Neural networks can usually lose most of their weights without losing much accuracy, but the speed benefit depends on where the zeros fall. If zeros are scattered at arbitrary positions ("unstructured" sparsity), accuracy holds up best, yet GPUs rarely run faster, because finding and gathering the surviving weights costs more than the skipped arithmetic saves. If zeros follow a rigid hardware pattern such as 2:4, where every group of four consecutive weights keeps exactly two, recent NVIDIA GPUs run the matrix multiply up to twice as fast. In language and vision models, switching from scattered to patterned zeros costs very little average accuracy. This proposal asks whether that cheap switch is equally cheap for the rare and transitional cell states that single-cell biology most needs to resolve. Fibroblasts are the clearest test case. They are not one cell type but a spectrum. It includes rare, short-lived, disease-driving states, such as inflammatory myofibroblasts that appear around day 7 after wounding, alongside abundant homeostatic states.
Our claim is that the accuracy price of hardware alignment is not spread evenly. It falls mostly on rare and transitional fibroblast states, and standard averaged metrics such as annotation accuracy and integration scores hide it. If that is true, it has two consequences. Compressed single-cell foundation models could quietly lose exactly the disease-relevant states they are used to find. And the fix should be cheap: choose the cells used to calibrate the pruned model so that rare states are represented, rather than giving up the hardware pattern. We will measure the per-state price of alignment, test whether state-balanced calibration recovers it, and report measured wall-clock speedups at the matrix shapes these biological models actually use. Those speedups have never been characterized.
Core Claim
Define the price of alignment for a cell state $s$ as $\Delta_s = R_s^{\text{unstr}} - R_s^{2:4}$. Here $R_s$ is the recall of state $s$ in downstream annotation from a compressed model's embeddings. "unstr" denotes unstructured pruning and "2:4" denotes hardware-patterned pruning, both at 50% sparsity.
We hypothesize that $\Delta_s$ grows as the state becomes rarer and more transitional. The mechanism is that rare states contribute little to the activations used to decide which weights to keep. Weights that matter only for those states therefore look unimportant. The rigid two-of-four choice is then forced to drop them, whereas unstructured pruning could keep them by sacrificing redundant weights elsewhere.
The proposed remedy keeps the 2:4 core but changes what the pruned layers are asked to reconstruct. We weight the calibration objective by inverse state frequency:
$$\min_{\hat W} \sum_{c} w_{s(c)} \,\lVert W x_c - \hat W x_c \rVert_2^2 ,$$
where $W$ is a dense layer, $\hat W$ is its compressed version with a 2:4 core, $x_c$ is the layer's input activation for calibration cell $c$, $s(c)$ is that cell's annotated or clustered state, and $w_s$ is proportional to the inverse frequency of state $s$.
Prior Work and Where This Sits
The biological motivation comes from atlas work showing that the important fibroblast distinctions are fine-grained and partly transitional. A single-cell and spatial genomics atlas of human skin fibroblasts integrates 2.1 million skin cells and defines subtypes F1–F8. Of these, F2, F3, F6 and F7 are conserved across about 5.8 million cells from seven tissues. A wound time course shows F6 inflammatory myofibroblasts peaking near day 7 and giving way to F7 by day 30. That gives us labeled common, rare, and transitional states, plus a trajectory, to test against.
The PDAC atlas in Integrating multi-modal transcriptomics separates a tumor-promoting POSTN+ fibroblast axis from a protective immune axis. The ccRCC review Cancer-associated fibroblasts in clear cell renal cell carcinoma documents that ablating the stroma wholesale backfires because it removes protective fibroblast subsets. These papers motivate the evaluation: what matters is whether a model still separates subsets that look similar but have opposite functions. They supply no method.
On the compute side, three papers supply the methods we reuse:
- SR-STE shows 2:4 training costs almost nothing on ImageNet: 77.0% top-1 versus 77.3% dense for ResNet-50.
- ARMOR wraps a 2:4 core in small block-diagonal correction matrices, $\hat W = A(W' \odot M)B$. It reports 39.51 on GPQA for Qwen 2.5-32B against 38.84 dense, and a 1.57× layer speedup.
- SMaT shows unstructured multiplies can beat dense cuBLAS on Tensor Cores above about 78% sparsity, after reordering rows.
All of these evaluate average accuracy on language, vision, or scientific matrices. None asks which subpopulations of inputs bear the loss. We borrow their compression methods and kernels, and change only the evaluation unit (per cell state) and the calibration data.
The closest work at the intersection uses "sparsity" in a different sense:
- DEPF prunes members of an ensemble of clusterings. It never touches network weights.
- Sparse autoencoders on single-cell foundation models imposes sparsity on activations to find interpretable features. It runs dense kernels and never compresses the model.
- scGPT skips zero-count genes in the input, cutting fine-tuning time by up to 60%. Its attention and feedforward weights stay dense.
We use the sparse-autoencoder features as a mechanistic probe rather than as a competitor. Calibration-Preserving Pruning shares our concern that compression can break reliability in ways averages hide. It targets calibrated uncertainty rather than subpopulation recall.
Research Program
We will compress scGPT's transformer layers to 50% sparsity under four regimes:
- Unstructured: activation-aware magnitude pruning.
- Naive 2:4: magnitude-based choice of two of every four weights.
- ARMOR 2:4: ARMOR with a generic calibration set, meaning cells sampled uniformly from CELLxGENE-like data.
- State-balanced ARMOR 2:4: ARMOR whose block-coordinate updates minimize the weighted objective above, using the same number of calibration cells.
Each regime is evaluated with and without a short recovery fine-tune on a fixed budget.
For each compressed model, we embed the held-out evaluation data. We then measure four things:
- Per-state recall of a fixed linear or k-nearest-neighbor classifier trained on dense-model labels.
- Trajectory fidelity: rank correlation of pseudotime with dense-model pseudotime on the wound time course, and whether the F6 day-7 peak is still recovered.
- Feature survival: the fraction of sparse-autoencoder features that are selective for fibroblast states and that still fire on the same cells after compression. This tests whether lost recall reflects lost internal features.
- Hardware cost: measured wall-clock throughput and energy per million cells embedded on Ampere/Hopper GPUs. We use vendor 2:4 kernels for the patterned models. We test unstructured models both as masked dense multiplies and with an SMaT-style reordered kernel.
scGPT's 512-wide layers are far smaller than the language-model layers where ARMOR measured 1.57×. We expect smaller realized speedups. Whether any speedup survives at these shapes is itself an open measurement. We found no reported result on Lacuna for 2:4 latency on single-cell foundation models.
For state labels, calibration weighting uses only training-split atlas annotations. A second variant uses unsupervised Leiden clusters, to check that the method does not depend on curated labels.
Baselines
Dense model (do nothing). The dense scGPT is the reference point for both accuracy and speed. scGPT reports 0.856 cell-type annotation accuracy on the Multiple Sclerosis dataset and AvgBIO up to 0.924 for integration (scGPT). We rerun it on our splits and hardware so every comparison shares data and GPUs. A compressed model "matches" dense if per-state recall for F3, F6 and F8 stays within the variation across three random seeds of the dense classifier.
Unstructured pruning. This is the accuracy ceiling for 50% sparsity. It is the regime the overhead literature says rarely runs faster on standard kernels (dynamic sparse reparameterization). We give it its best chance on speed with an SMaT-style kernel. SMaT's reported crossover with cuBLAS is at 78% sparsity, so we expect no speedup at 50%. We will report what we measure.
ARMOR with generic calibration. This is the strongest existing hardware-aligned competitor. The reported numbers are language-only: GPQA 39.51 versus 38.84 dense on Qwen 2.5-32B, nearly halving the perplexity gap on Llama-2-13B, and 1.57× layer speedup (ARMOR). There is no reported result on Lacuna for ARMOR on single-cell models. It gets the same calibration cell count, block size, and optimization iterations as our variant.
Naive 2:4 and SR-STE-style recovery fine-tune. These test whether plain retraining closes the gap. Fine-tuning uses the identical step budget across regimes.
Ablations. Two controls isolate the key ingredient, which is the calibration weighting. A random-reweighting control uses the same weight distribution assigned to random cells, ruling out "any reweighting helps." A uniform-but-larger calibration set, with four times as many cells and no weighting, rules out "more data helps."
State-balanced ARMOR beats the field if two things hold. It must reduce the rare-state price of alignment $\Delta_s$ for F6/F3/F8 by at least half relative to generic ARMOR. It must also leave common-state recall and wall-clock throughput unchanged.
Why not just…?
Why not just run the dense model? For a one-off analysis, that is a fine answer, and we report it honestly. The case for compression rests on repeated encoding of atlases that already reach 5.8 million cells, and on interactive query mapping. If measured 2:4 throughput at scGPT shapes stays near 1×, the dense model wins on cost and the hardware motivation collapses. The per-state finding would then still matter for anyone compressing larger single-cell models. Our speed measurements settle this directly.
Why not just use unstructured pruning with a good sparse kernel? At 50% sparsity, the available evidence says the indexing overhead dominates. Software sparse kernels spend up to 40% of execution time on index reconstruction and memory bank conflicts (Celty). SMaT only beats dense above about 78% sparsity. Pushing unstructured sparsity to 80%+ to reach that regime would likely cost rare-state accuracy too. We include an 80% unstructured run to check this trade-off rather than assume it.
Why not just fine-tune after 2:4 pruning on ordinary data? This is the real threat. A short recovery fine-tune might restore rare states without any special calibration. We give fine-tuning the same budget and also cross it with state-balanced calibration. If uniform fine-tuning alone closes $\Delta_s$, the contribution shrinks to the diagnostic finding plus a recommendation. We would say so.
Why not just use low-rank compression instead? Low-rank factors restrict each layer to one shared subspace, which may be exactly where rare-state directions get dropped. The sparse-plus-low-rank results in 3BASiL and ARMOR favor hybrids over pure low rank. We run a parameter-matched singular-value-decomposition baseline under the same per-state evaluation to confirm.
Decisive Checks
The central claim is falsified if, on the skin atlas and its cross-tissue extension, $\Delta_s$ for rare states (F3, F6, F8) is no larger than for abundant states (F2), within seed variance. Two further results strengthen or weaken the remedy. If state-balanced calibration recovers rare-state recall where random reweighting and larger uniform calibration do not, the weighting is doing real work. If the lost sparse-autoencoder features track the lost recall, the mechanism is internal feature loss rather than classifier noise.
External validity comes from two transfers. First, we apply the skin-calibrated compressed model to the PDAC atlas without recalibration and check whether POSTN+ fibroblasts remain separable from other fibroblasts. Second, we test on the DEPF colorectal (GSE144735) and HNSCC fibroblast subsets, using DEPF's NMI-based clustering protocol. Finally, the wound time course tests whether compression flattens the F6 day-7 peak. A biologist would notice that failure even when the averaged benchmarks look fine.
References
- A single-cell and spatial genomics atlas of human skin fibroblasts reveals shared disease-related fibroblast subtypes across tissues
- Integrating multi-modal transcriptomics identifies cellular subtypes with distinct roles in PDAC progression
- Cancer-associated fibroblasts in clear cell renal cell carcinoma: functional heterogeneity, tumor microenvironment crosstalk, and therapeutic opportunities
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- High Performance Unstructured SpMM Computation Using Tensor Cores
- Reliable Identification and Interpretation of Single‐Cell Molecular Heterogeneity and Transcriptional Regulation using Dynamic Ensemble Pruning
- Sparse Autoencoders Reveal Interpretable Features in Single-Cell Foundation Models
- scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
- Calibration-Preserving Pruning: Compression as a Reliability Contract
- Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
- Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs