When a stem cell population ages, the usual summary numbers can look unchanged while the cells that matter most quietly disappear. Young and aged adipose stem cells match on frequency, viability and growth, yet single-cell profiling shows that aging wipes out a small pro-vascular subpopulation expressing Hif1a, Vegfa and Cxcl12. That loss is enough to cut tubule formation from 11.4 to 3.1 tubules per high-power field (Aging disrupts cell subpopulation dynamics and diminishes the function of mesenchymal stem cells). Compressing neural networks has a similar failure pattern. Quantization stores a model's weights with fewer bits, for example 2 bits per weight instead of 16. Below about 3 bits, language models keep their fluency while their reasoning collapses (Evaluating Quantized Large Language Models). This proposal asks whether the same thing happens to single-cell foundation models, which are transformers pretrained on millions of cell expression profiles. When such a model is compressed, does it keep the broad cell types and lose the rare, depleted and in-between cell states that define tissue aging? And can we stop this without labels?
The core claim is twofold. First, weight quantization of single-cell transformers erases rare and transitional cell states at higher bit-widths than it erases major cell types, and aggregate scores such as overall annotation accuracy or average biological conservation will hide this. Second, the cause is fixable. Standard post-training quantizers estimate which directions in activation space matter from a calibration set, and that set is dominated by abundant cells. We propose to reweight the calibration statistics by inverse cell density in the full-precision model's own embedding space. This requires no labels, and it should recover rare-state fidelity at 2 to 3 bits at almost no cost to common-state accuracy.
Why this is open
No paper in our search has quantized the weights of a single-cell foundation model. The existing "quantization" work in single-cell biology quantizes data rather than models. MetaQ vector-quantizes expression vectors into metacells, CVQVAE learns a discrete latent codebook for multi-omics integration, and QOT quantizes cell distributions to speed up optimal transport distances. None of the three measures how low-precision weights distort what a trained transformer represents. scGPT only discretizes its inputs, as binned expression values, and runs in floating point. The zero-shot audit in Zero-shot evaluation reveals limitations of single-cell foundation models evaluates full-precision models only.
The quantization literature supplies our methods and also constrains our designs. QuIP rotates weights with random orthogonal matrices so that outliers are spread out, then rounds layer by layer. Its rounding minimizes a proxy error
$$E = \mathrm{Tr}\big((W-\hat W)\,H\,(W-\hat W)^\top\big), \qquad H = \textstyle\sum_i x_i x_i^\top,$$
where $W$ is a layer's weight matrix, $\hat W$ its quantized version, and $H$ the second-moment matrix of the layer's input activations $x_i$ over calibration tokens. CRVQ protects the roughly 2% of channels that this same Hessian marks as most important. Both treat the calibration set as given. That is exactly where rare cells lose out: a cell type making up 1% of calibration cells contributes about 1% of $H$, so rounding error gets pushed into the directions that only rare cells use. The file names calibration dependency as an open problem (AdpQ and AlphaQ remove calibration entirely). No paper has examined how the composition of the calibration set interacts with population structure.
Core method
The change itself is small, and its smallness is deliberate. We run the full-precision model once over a calibration pool and embed each cell. For each cell $c$ we estimate a local density $\rho_c$ using its $k$-nearest-neighbour distance in that embedding. We then replace $H$ with
$$H_w = \sum_c \rho_c^{-\alpha} \sum_{t \in c} x_t x_t^\top,$$
where $t$ runs over the gene tokens of cell $c$. The exponent $\alpha \in [0,1]$ moves between the standard estimate ($\alpha=0$) and fully density-flattened weighting ($\alpha=1$). $H_w$ is passed unchanged into GPTQ-style rounding, QuIP's rotation-plus-rounding, or CRVQ's channel selection. This last point matters because density weighting also changes which channels CRVQ treats as important. That gives us a secondary analysis: we can check whether the channels promoted by density weighting carry the aging-associated programs described in the file, such as Sod2/Sod3 and the Hif1a/Vegfa/Cxcl12 module.
We need a way to measure "rare-state fidelity" that a single accuracy number cannot hide, and we score three things per cell state. The first is rare-state recall: k-nearest-neighbour label transfer recall restricted to the rarest states. The second is snapping rate, defined for transitional cells. These are cells partway between two fates, for example the lung's pre-alveolar type 1 transitional state (PATS), in which alveolar type 2 stem cells stall on their way to becoming type 1 cells. The snapping rate is the fraction of such cells whose quantized-embedding neighbours move to one endpoint fate. The third is pseudotime rank correlation along the type 2 → PATS → type 1 path, compared between full-precision and quantized embeddings.
Research Program
The work has three stages. First, we map the "biological performance cliff." We quantize scGPT at 8, 4, 3 and 2 bits per weight using round-to-nearest, GPTQ/OPTQ-style rounding, QuIP, and a CRVQ-style channel relaxation. We then plot per-state fidelity against bit-width, broken down by state frequency. The hypothesis is confirmed if rare and transitional states degrade at a strictly higher bit-width than common ones while aggregate accuracy stays flat.
Second, we test whether density-weighted calibration moves that cliff. We also test whether it does so better than the simpler fixes listed below.
Third, we add a controlled "synthetic aging" experiment that separates rarity from biology. In a large atlas, we downsample a chosen cell type to 10%, 1% and 0.1% of its natural abundance, both in the evaluation set and in the calibration pool. This mimics the selective depletion seen in aging (Aging disrupts cell subpopulation dynamics and diminishes the function of mesenchymal stem cells; MicroRNA-21 maintains hematopoietic stem cell homeostasis). It lets us ask whether quantization damage scales with calibration frequency, holding the cell type fixed.
Baselines
The naive reference point is the full-precision (FP16) model. Every quantized variant is scored against it on identical cells, and we report the fraction of FP16 rare-state recall retained. Alongside it we place the "do not use a foundation model" floor: highly variable gene selection, scVI and Harmony. The zero-shot audit found these lightweight methods often beat unquantized scGPT and Geneformer (Zero-shot evaluation reveals limitations of single-cell foundation models). If a quantized model falls below this floor on rare states, compression has made it pointless for this purpose, and we will say so.
The strongest existing compression method is QuIP, with CRVQ-style relaxation as the strongest variant for protecting important channels. We run both with their published procedures. They use the same calibration pool size, the same number of calibration cells, the same group size and the same bits per weight as our method. They differ from ours only in the weighting of $H$. Each method gets the same hyperparameter budget, a fixed grid over damping and group size selected on a held-out validation split.
On language models these methods report the following. QuIP reaches WikiText-2 perplexity 6.33 at 2 bits on Llama 2 70B, against 123.91 for OPTQ (QuIP). CRVQ reaches 9.81 at about 1.06 bits on LLaMA2-13B, against 15.25 for AQLM (CRVQ). There is no reported result on Lacuna for any of these methods on single-cell foundation models. The full-precision anchors therefore come from scGPT's own evaluation: annotation accuracy 0.856 on the multiple sclerosis dataset, and AvgBIO up to 0.924 on batch integration (scGPT).
The key ablation isolates the role of density reweighting by comparing it with two alternatives. The first is class-balanced calibration, which uses ground-truth labels to equalize cell types. The second is a calibration-free quantizer in the style of AdpQ. Beating QuIP means recovering, at 2 or 3 bits, a statistically significant share of the rare-state recall and pseudotime correlation that QuIP loses, across five quantization seeds. In addition, common-state accuracy must drop by no more than 0.5 points relative to QuIP.
Why not just…?
Why not just quantize to 4 bits, which is near-lossless? Four-bit quantization is near-lossless for language models (Renren Jin et al.). That evidence measures aggregate scores, and aggregate scores are exactly what we argue hides rare-state loss. Whether 4 bits is safe for transitional lung cells has never been measured. If our sweep shows that 4-bit models keep rare-state fidelity, that is a useful negative result: it would give the field a safe default. The density-weighting method would then matter only for the more aggressive 2–3 bit regime.
Why not just use per-class metrics like macro-F1? Per-class metrics are one of our tools, but they need labels, and the states that matter in aging are often unlabeled or newly found. PATS, for example, was found as an unlabeled cluster (COVID-19 tissue atlases). Metrics also diagnose the problem without fixing it. The snapping-rate and pseudotime measures target continuous states that have no class boundary to score.
Why not just balance the calibration set by cell type? This is the real threat. When labels exist, class balancing may match density weighting. We include it as a head-to-head ablation and report it honestly. Our method's claim is narrower: it achieves comparable recovery without labels and also helps unlabeled transitional states. A further check is whether density weighting beats class balancing on PATS when PATS is deliberately left out of the label set.
Why quantize a model this small at all, rather than use scVI? scGPT fits in FP16 memory, so deployment savings are modest today. Model scale is growing, though (Assessing Scale and Predictive Diversity in Models for Single-Cell Transcriptomics based on Geneformer). The question also generalizes: compression of any biological model should be audited on the states that carry the biology. We evaluate in the fine-tuned and perturbation-prediction settings where scGPT beat specialized methods, and we report the scVI floor throughout.
Proposed Experiments
The main lung test uses the COVID-19 lung atlas of 106,792 nuclei, with PATS identified as cluster 7 (COVID-19 tissue atlases reveal SARS-CoV-2 pathology and cellular targets). There we measure PATS snapping rate and pseudotime correlation under each quantizer and bit-width.
The synthetic-depletion experiment runs on Tabula Sapiens (483k cells) and the Immune 330k benchmark from the zero-shot audit. Annotation transfer is tested on the multiple sclerosis dataset, with FP16 accuracy 0.856. Perturbation prediction on Adamson and Norman Perturb-seq tests whether rare-state protection costs generative quality.
The murine adipose stem cell cohort is a small, targeted check. It covers about 70 genes across hundreds of cells, so it can only show whether quantized embeddings still separate the pro-vascular cluster in young versus aged cells. We will not overinterpret it.
The decisive checks are three. First, rare-state fidelity must fall at a higher bit-width than overall accuracy. If the two curves coincide, the core claim is false. Second, in the synthetic-depletion experiment, damage must track calibration frequency for a fixed cell type. If it does not, calibration composition is not the mechanism, and the method has no rationale. Third, density weighting must beat QuIP and match class-balanced calibration without using labels, and it must beat class balancing on held-out unlabeled transitional states.
References
- Aging disrupts cell subpopulation dynamics and diminishes the function of mesenchymal stem cells
- Evaluating Quantized Large Language Models
- MetaQ: fast, scalable and accurate metacell inference via single-cell quantization
- CVQVAE: A representation learning based method for multi-omics single cell data integration
- QOT: Efficient Computation of Sample Level Distance Matrix from Single-Cell Omics Data through Quantized Optimal Transport
- scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
- Zero-shot evaluation reveals limitations of single-cell foundation models
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs
- AdpQ: A Zero-shot Calibration Free Adaptive Post Training Quantization Method for LLMs
- AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
- MicroRNA-21 maintains hematopoietic stem cell homeostasis through sustaining the NF-κB signaling pathway in mice
- A Comprehensive Evaluation of Quantization Strategies for Large Language Models
- COVID-19 tissue atlases reveal SARS-CoV-2 pathology and cellular targets
- Assessing Scale and Predictive Diversity in Models for Single-Cell Transcriptomics based on Geneformer