Packing Single-Cell Count Matrices onto Tensor Cores

In single-cell RNA sequencing, each cell is a row of roughly 20,000 gene counts, and 80–95% of those counts are zero. Some zeros are real (the gene is off). Many are measurement failures, called "dropouts," where a transcript was present but never captured. Every neural model of these data starts the same way: it multiplies this mostly-zero cell-by-gene matrix by a dense weight matrix. GPUs handle that step badly. Dense libraries waste work on the zeros. General sparse libraries are tuned for scientific matrices that are more than 99% empty, and at the 80–95% sparsity of real cell data their bookkeeping costs more than the arithmetic they skip. So practitioners either throw away most genes before modeling, or, like scGPT, drop zero genes token by token and pay for ragged, padded batches.

This note proposes that the zeros in single-cell data are not scattered the way GPU kernels assume, and that their structure is predictable from biology. Cells of the same type detect largely the same genes, and genes in the same program are detected together. If genes are ordered once by co-detection module, and each 16-cell slice of a minibatch is drawn from a single coarse cell cluster, the input matrix becomes close to block-dense. It can then run on Tensor Cores, the GPU's dense matrix-multiply units, in the blocked format that SMaT showed can beat both cuSPARSE and dense cuBLAS. The difference from SMaT is where the ordering comes from. SMaT finds its ordering by an expensive search that only pays off for a fixed matrix. Here the ordering costs almost nothing, so it still works during training, when minibatches are reshuffled every step.

Core Claim

The sparse operand in single-cell learning is the data, not the weights, and its sparsity pattern is mostly explained by two factors: cell type and gene detection rate. The claim has three parts, each of which can be tested.

First, ordering genes by co-detection module and grouping cells by cluster, at the granularity of one Tensor Core tile, raises block fill rate far above random ordering. Block fill rate means the fraction of each stored 16×8 block that holds nonzeros. We expect fill to come close to what SMaT's offline Jaccard reordering achieves, at a small fraction of its preprocessing cost.

Second, the physics of dropout tells us how to split the genes. A gene's chance of being detected rises with its mean expression; under Poisson-like capture it is roughly $1-e^{-c\mu}$, with $\mu$ the gene's mean expression and $c$ the capture efficiency. Genes detected in nearly every cell should go through a dense matrix multiply. Genes in the middle range carry the cell-type-specific structure and should run in the blocked Tensor Core format. Rare-detection genes, which are dominated by dropout and have no block structure, should run in a compressed-row sparse format.

Third, this three-tier kernel makes the first layer and its weight gradient exact and cheap enough on the full gene panel that it costs about as much as a dense model restricted to highly variable genes (HVGs).

Restricting to HVGs is the practical alternative this idea has to beat. The point is not to discard genes; it is to keep them at roughly HVG-level cost.

Batch diversity is preserved by construction. Only the 16 rows inside one tile need to be homogeneous. A 512-cell minibatch is 32 tiles drawn from different clusters, sampled in proportion to cluster size. The batch is therefore a stratified sample rather than a biased one, and the multiply itself is exact.

Research Program

Measurement study. The first piece of work is a measurement nobody has published: the block structure of real transcriptomic matrices. Using the five-tissue benchmark from Zero-shot evaluation reveals limitations of single-cell foundation models (Pancreas 16k, PBMC 12k, PBMC 95k, Immune 330k, Tabula Sapiens 483k) and the 600,000-cell Human Cell Landscape used by SERM, we compute block fill rate under five orderings: native file order, random, SMaT's Jaccard row reordering, our module-and-cluster ordering, and an oracle that clusters with ground-truth cell-type labels.

Gene ordering. Genes are ordered once per reference corpus. We cluster a binary co-detection matrix estimated from a subsample of a few thousand cells, in the spirit of DECODE, which showed that presence/absence patterns form an informative gene dependency network. We need only a coarse ordering, not DECODE's statistical tests or imputation.

Cell grouping. Cells are grouped by a cheap clustering step in a low-dimensional projection, done once and reused across epochs. For new query cells, which arrive without labels, a nearest-centroid assignment against the reference clusters replaces the clustering. The detection-rate thresholds that define the three gene tiers are chosen by a per-dataset sweep and reported.

Kernel. We build the three-tier kernel for two products: the forward multiply of the cell-by-gene matrix with the gene-by-hidden weights, and the transposed product that gives the weight gradient. The blocked tier reuses SMaT's blocked-row layout, asynchronous loads and double buffering. The new parts are the zero-search ordering, the tiling-aware minibatch sampler, and the tier split.

Models. We plug the kernel into two models that use the full gene panel. One is a negative-binomial variational autoencoder in the scVI style. The other is a gene-embedding input stage of the kind used by scGPT and Cell-JEPA. In that stage each cell's input is a count-weighted sum of gene embeddings, which is exactly this sparse-times-dense product. Cell-JEPA's authors note that its 28M-parameter scale was limited by compute, which motivates the second model.

We will be honest about Amdahl's law: a likelihood decoder that scores all genes remains dense. We therefore report kernel-level and end-to-end speedups separately.

Baselines

Naive reference. The do-nothing reference is the matrix densified and multiplied with cuBLAS in half precision. Its cost does not depend on sparsity, which makes it the floor any sparse method must clear.

Sparse references. There are two. The first is cuSPARSE compressed-row multiply on the same matrix. The second is a gather-and-sum embedding-bag implementation, which is what scGPT-style zero-gene omission amounts to computationally. It is a serious competitor because it does no wasted arithmetic and needs no reordering. Its weakness is that it rereads each weight row for every nonzero instead of reusing it across the cells in a block.

Strongest existing method. SMaT itself, given full offline Jaccard reordering of the exact matrix it will multiply. This is the upper bound on what generic reordering achieves when its cost is ignored. On SuiteSparse, SMaT averaged 2.6x faster than DASP, the second-best library, was up to 125x faster than cuSPARSE, and beat cuBLAS at sparsity as low as 78% on synthetic band matrices (SMaT). There is no reported result on Lacuna for SMaT or any Tensor Core sparse kernel on transcriptomic matrices.

Fairness. All kernels run on the same Ampere-class GPU, in FP16 with FP32 accumulation, on the same matrices and hidden widths (128 to 1024). Each kernel's tile and scheduling parameters get the same autotuning budget. Preprocessing time is reported separately for every method and also folded into amortized time per training epoch.

Ablations. Ablations isolate each ingredient: gene ordering alone with random cell tiles, cell tiling alone with random gene order, and the blocked kernel without the tier split. We also keep the module-and-cluster ordering but run SMaT's own row ordering inside each shuffled batch, which isolates whether the biological prior, rather than the kernel, is what matters.

What counts as success. We count it as a win if amortized per-epoch time for the full-gene first layer beats both cuBLAS and the embedding bag by at least 1.5x at the observed sparsity, and comes within 20% of SMaT-with-oracle-reordering. Downstream quality must also be statistically indistinguishable from random-batch training.

Why not just…?

Why not just select 2,000 highly variable genes and use a dense model? This is the most serious threat. The zero-shot evaluation paper found that HVG selection beat scGPT and Geneformer on zero-shot clustering and gave the best batch integration. If full-gene models never beat HVG models on quality, a faster full-gene kernel has no customers beyond foundation-model training. We address this directly. We train matched full-panel and HVG versions of the same autoencoder under equal wall-clock budgets. We evaluate overall clustering and also recovery of rare cell types and transcription-factor-driven states, where the single-cell analysis literature expects HVG filtering to lose signal (Challenges and emerging directions in single-cell analysis). If the full panel shows no advantage anywhere, we report the kernel result as relevant only to all-gene foundation models.

Why not just run SMaT with its own reordering? For a fixed atlas embedded once, offline Jaccard reordering is amortized, and SMaT may simply win. Our case rests on training and query settings, where the rows of each multiply change every step and SMaT would have to reorder every batch. Two experiments settle this: preprocessing time per batch, and the ablation running SMaT's ordering inside shuffled batches. If the biological ordering gives much lower fill than Jaccard, the prior is not doing the work, and the honest conclusion is that generic reordering should be made cheaper instead.

Why not just use the embedding-bag gather that scGPT effectively uses? It needs no reordering and wastes no arithmetic, so it may already be near optimal when the hidden width is small. We sweep hidden width and expect blocked reuse to win only above some width. We will report that crossover rather than assume it.

Why not just prune the model's weights to the 2:4 pattern, two nonzeros in every four, that sparse Tensor Cores accelerate? Learning N:M sparse networks from scratch gives about 2x on supported hardware. But that targets weight sparsity, which is orthogonal to input sparsity. It also risks accuracy in compact models: Channel Permutations for N:M Sparsity reports a 4.01% drop on SqueezeNet without permutation. The two could be combined later. On its own it leaves the 80–95% input sparsity unexploited.

Proposed Experiments and Decisive Checks

Check 1: fill rate. The first decisive check is block fill rate on the five-tissue benchmark and the Human Cell Landscape. If module-and-cluster ordering does not at least double fill relative to random ordering, and reach most of the oracle-label ordering's fill, the premise fails and we stop. Our expectation, from dropout physics, is that fill concentrates in the middle detection tier. If instead fill is spread evenly across tiers, the tier split is unnecessary, and we will report that.

Check 2: kernel timing. We time the forward and weight-gradient products against all baselines across sparsity levels obtained by subsampling read depth. This traces where the method crosses over dense cuBLAS on biological matrices, to be compared with the 78% crossover SMaT reported for synthetic matrices.

Check 3: end-to-end training. We train the scVI-style autoencoder and the gene-embedding encoder on Tabula Sapiens and the Human Cell Landscape with random versus tile-stratified batching. Quality is measured as clustering normalized mutual information and adjusted Rand index (NMI, ARI), the integration score AvgBIO, and batch-mixing scores on the five-tissue benchmark. The reference point is scGPT's reported integration AvgBIO of up to 0.924 (scGPT), alongside the 60% fine-tuning time reduction scGPT obtained from zero-gene omission. Quality must not drop under stratified batching.

Check 4: query mapping. For new cells projected onto a reference, the fixed gene order plus nearest-centroid tiling must keep its speedup without any per-query reordering. If it does not, the method's advantage collapses to the training setting alone.