Single-cell foundation models are Transformers trained on tens of millions of cells' gene-expression profiles. Compressing them to 4 or 3 bits per number, a step routine in vision, could quietly change their biological conclusions. The usual benchmarks would not catch this. A quantized model might still tell a macrophage from a T cell, because those cell types sit far apart in the model's internal representation. It might nonetheless lose the much smaller differences that separate an inflammatory macrophage from a pro-repair one, or a cell before treatment from the same cell after. Those small differences are what macrophage-reprogramming research measures. That work tracks cells pushed from a destructive "M1" state toward a regenerative "M2" state by drugs, stem-cell secretions, or nanoparticles. This note proposes to measure that failure and diagnose its cause. If the cause is clear, a cheap fix aimed at it may follow.
The proposal makes quantization the core mechanism and macrophage polarization the test regime. The aim is not edge deployment of sequencing analysis, which is a weak motivation since sequencers are not bedside devices. The aim is a plain question: how much numerical precision does a continuous, fine-grained immune state need inside a Transformer? Low-bit quantization is a controlled way to remove precision and watch what disappears first.
Core Claim
We claim that standard post-training quantization (PTQ), which rounds a trained model's weights and activations to low-bit integers using a small calibration set and no retraining, hurts fine-grained macrophage state structure much more than coarse cell-type accuracy. We predict that at 4 bits, coarse annotation stays near full precision while the ordering of macrophages along an M1-to-M2 axis is already badly scrambled. If so, reporting cell-type accuracy alone would certify quantized biological models as safe when they are not.
We further hypothesize that the damage comes mostly from two failure modes documented in vision Transformers. The first is a handful of extreme activation outliers that inflate the rounding step for everything else. ADFQ-ViT found such post-LayerNorm patch outliers. The second is split, two-peaked activations after the Key projections, which PTQ4SAM found in Segment Anything. Low-expressed regulatory genes would be the first casualties: they land in the bottom expression bins, and a coarse rounding step flattens them. The transcription factors IRF4 and KLF4, which the artesunate study places at the center of M2 polarization, are natural sentinels for this.
Research Program
The model is scGPT, from scGPT. It is a generative Transformer pre-trained on 33 million CELLxGENE cells, with gene tokens, binned expression values, and attention maps the authors read as gene-regulatory interactions. It was evaluated only in floating point. SCTrans, from SCTrans, serves as a second architecture so that findings are not specific to one model.
The first step defines a polarization axis that does not depend on the model. We take the stimulated M1 versus M2 bulk contrasts that the artesunate study drew on (GSE86298, GSE99296, GSE106700, GSE158094). From these we derive an M1 gene signature and an M2 gene signature. We then score macrophages in the human liver atlas of MacParland et al.. That atlas separates an inflammatory LYZ+/S100A8+ Kupffer cluster from an immunoregulatory MARCO+/VSIG4+ cluster, and the clusters overlap rather than forming a clean binary. The signature score gives every cell a continuous position along the axis that is independent of the model.
Inside the full-precision model, we fit a linear probe from cell embeddings to that score. The probe defines a direction $u$ in embedding space. For a quantized model we then compute the axis distortion
$$D = \frac{\mathrm{median}_i \, |u^\top (z_i^{q} - z_i)|}{\mathrm{IQR}_i(u^\top z_i)},$$
where $z_i$ and $z_i^{q}$ are cell $i$'s full-precision and quantized embeddings and IQR is the interquartile range of positions along $u$. The ratio compares how far quantization moves cells along the axis with how spread out the cells are along it. We also report the Spearman rank correlation of cell ordering along $u$ before and after quantization. As a contrast, we compute the same distortion along the direction separating myeloid from lymphoid cells. The core claim predicts $D_{\text{polarization}} \gg D_{\text{lineage}}$ at equal bit-width.
The second step is diagnosis. At each bit-width we quantize one component family at a time: the Q/K/V projections, the MLP, the Softmax output, and the post-LayerNorm activations. Each time we record which family drives $D$. We also log activation histograms, looking for outlier tails and two-peaked Key channels. Separately, we check how the attention maps' gene-gene edges for IRF4, KLF4, EREG, and CSTA change. Those four genes come from the artesunate and diabetic foot ulcer studies. Q-MHSA reported that quantization erases sparse, high-magnitude attention values, and these edges test whether the same happens to scGPT's regulatory readout.
The third step is a targeted fix, conditional on the diagnosis. It is axis-preserving calibration: the standard block-wise reconstruction objective plus a term $\lambda \sum_i (u^\top (z_i^{q} - z_i))^2$, where $\lambda$ sets the term's weight. The fix borrows the reconstruction machinery from FIMA-Q, which approximates the Fisher information with a diagonal plus a low-rank term, and from ADFQ-ViT's per-patch outlier handling, applied here per gene token. The fix is secondary. If the diagnosis shows that one existing quantizer already repairs the axis, we report that and drop the new loss.
Baselines
The naive reference points are the full-precision model and plain round-to-nearest uniform quantization at W8A8, W6A6, W4A4, and W3A3 (weights/activations in bits). Round-to-nearest shows the uncorrected failure, and full precision is the ceiling.
The strongest existing competitors are the vision PTQ methods, ported unchanged to scGPT. FIMA-Q is the strongest general method; it reports 64.09% top-1 on ViT-S at W3/A3 on ImageNet-1K, against roughly 54% for DopQ-ViT. ADFQ-ViT is the most direct match to the outlier hypothesis; it reports a 10.23% top-1 gain over RepQ-ViT on 4-bit ViT-B. PTQ4SAM's sign-flipping fix for two-peaked Key activations is the most direct match to the second hypothesized failure; it reports 32.1 mAP on 4-bit SAM-L on MS-COCO. No reported result was found on Lacuna for any low-bit quantized single-cell Transformer.
Every method gets the same 1,024 calibration cells, the same three-seed protocol, and an equal grid-search budget over its hyperparameters. Every method is scored on the same held-out cells.
The ablation that isolates our one new ingredient keeps everything in axis-preserving calibration except the axis term. It uses macrophage-enriched calibration data with the plain reconstruction loss. Axis-preserving calibration "wins" only if it beats both this ablation and the best ported method by a margin outside the seed variance on $D$ and rank correlation, while keeping coarse annotation within the ported methods' range.
The coarse-task anchors come from the cited papers. scGPT reports up to 0.856 annotation accuracy on the multiple sclerosis dataset, and SCTrans reports 0.817 on Zheng68K.
Why not just…?
Why not just use the two-gene classical classifier? Cao et al. reached an AUC of 0.944 on an external diabetic foot ulcer cohort with EREG/CSTA and Firth regression. For binary bulk diagnosis, that classifier is the right tool, and we do not compete with it there. The question here is different. It is whether a foundation model's continuous state ordering and attention-derived gene links stay trustworthy under compression, and a two-gene score has neither. We will still fit the two-gene score and a LASSO model on the liver macrophages as a reference ordering. If either one reproduces the signature-derived axis as well as full-precision scGPT, that is a genuine finding about the value of the foundation model for this task, and we will report it.
Why not just stay at INT8? This is the strongest objection. ADFQ-ViT reports only a 0.67% average drop at 6 bits, and PTQ4SAM's W6A6 SAM-L matches full precision. If the axis survives at 6 to 8 bits, the practical advice is simply "use INT8". The study is designed to give exactly that advice if the data support it. The bit-width sweep locates the precision at which polarization structure breaks, relative to where coarse accuracy breaks. That threshold is the scientific result, whether it falls at 3 bits or 8.
Why not just compress the data with MetaQ? MetaQ groups similar cells into a learned codebook of "metacells", which reduces how many cells a model must process. It is orthogonal to quantization because it shrinks the number of cells, not the precision of the model. It does pose a related risk: averaging cells into metacells could blur the same fine axis. We will run it as a side comparison, computing the rank correlation along the polarization axis for MetaQ metacells versus single cells, so that the two kinds of compression are measured on one scale.
Why not just calibrate on macrophages? Choosing calibration data that matches the target cells may be enough by itself. That is why it is the central ablation rather than an afterthought. If it matches axis-preserving calibration, the method contribution collapses into a recommendation about how to choose calibration data, and the diagnostic study stands alone.
Decisive Checks
Three outcomes would decide the study.
The first is the core claim. At W4A4 with round-to-nearest, and with the best ported method, we expect coarse annotation on Zheng68K and the multiple sclerosis data to stay within a few points of full precision, while the polarization-axis rank correlation on liver macrophages falls well below the lineage-axis rank correlation. If the two axes degrade together, the core claim is false.
The second is diagnostic. If quantizing only the post-LayerNorm activations, or only the Key projections, accounts for most of $D$, and the matching ported fix restores the axis, that confirms the mechanism. It would also show that vision-derived quantizers transfer to transcriptomic Transformers.
The third tests whether the biology survives. On the Norman and Adamson Perturb-seq data, we compare predicted versus observed expression change after each perturbation between full-precision and quantized scGPT. On the liver data, we measure how stable the attention edges for IRF4/KLF4 are compared with edges for highly expressed housekeeping genes. Selective loss of the edges for low-expressed regulators, relative to housekeeping genes, would directly support the claim that quantization erases the genes that control macrophage polarization first.
References
- ADFQ-ViT: Activation-Distribution-Friendly post-training Quantization for Vision Transformers
- PTQ4SAM: Post-Training Quantization for Segment Anything
- Artesunate Inhibits Neointimal Hyperplasia by Promoting IRF4 Associated Macrophage Polarization
- scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
- SCTrans: Multi-scale scRNA-seq Sub-vector Completion Transformer for Gene-selective Cell Type Annotation
- Single cell RNA sequencing of human liver reveals distinct intrahepatic macrophage populations
- Adipose-Derived Mesenchymal Stem Cells Accelerate Diabetic Foot Ulcer Healing by Promoting Macrophage M2 Polarization Through Downregulation of EREG and CSTA
- A Universal Self-Attention Enhancement for Bridging Low-bit Quantization and Vision Transformers
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
- MetaQ: fast, scalable and accurate metacell inference via single-cell quantization