Two fields that never talk to each other share the same bottleneck. Speech recognition once needed thousands of hours of human-transcribed audio, until self-supervised pre-training changed the economics: wav2vec 2.0 masks spans of an audio recording and learns to pick, out of a set of decoys, the correct discrete token — drawn from a learned codebook of speech sounds — for the masked position; with that pre-training, 10 minutes of labeled audio sufficed to reach a 4.8% word error rate on clean test speech wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Tumor immunology has the mirrored problem. The immunosuppressive tumor microenvironment (TME) — the ecosystem of macrophages, T cells, and fibroblasts that excludes or disarms immune attack inside solid tumors — is now measurable at single-cell resolution, but the measurements are extremely noisy, and the clinically labeled cohorts are tiny: 23 annotated pancreatic slides A Novel Framework for Characterization of Tumor-Immune Spatial Relationships in Tumor Microenvironment, about 50 spatially profiled colorectal cancer patients Spatially resolved C1QC+ macrophage-CD4+ T cell niche in colorectal cancer microenvironment: implications for immunotherapy response, 129 ovarian cancer patients The MYBL2–CCL2 axis promotes tumor progression and resistance to anti-PD-1 therapy in ovarian cancer by inducing immunosuppressive macrophages.
The proposal is to import the speech recipe wholesale. Treat each cell as an audio frame: order cells into "utterances" by their physical position in tissue, quantize each cell's noisy molecular profile into a discrete functional-state token from a learned codebook, pre-train a masked model to identify the correct token among distractors drawn from the same tissue region, and then train a small adapter on top of the token sequences to predict immunotherapy response or physical immune-exclusion barriers from a handful of labeled patients. The analogy is not decorative. The exact failure wav2vec 2.0's quantization solves in audio — a noisy continuous signal that cannot be learned from scarce supervision — is documented as the central failure of current single-cell self-supervised models, which fit technical dropout noise instead of biology Cell-JEPA: Latent Representation Learning for Single-Cell Transcriptomics.
Core Claim
The claim is that vector quantization is the missing noise filter and reuse layer for tumor-microenvironment representation learning. Every existing self-supervised single-cell model keeps cells in a continuous latent space: mask-sc reconstructs masked expression entries Masked Modeling for Single-cell Clustering of scRNA‐seq Data, Cell-JEPA predicts a teacher's continuous embedding Cell-JEPA, and scJEPA predicts latents with a distributional regularizer scJEPA: Dual-Space Self-Supervised Learning for Single-Cell Representations. None quantizes profiles into a finite vocabulary of discrete functional states, and none is evaluated on spatial tumor architecture or clinical response. We hypothesize that a learned codebook does two things continuous latents cannot. First, it snaps dropout-corrupted profiles onto canonical states, so the pre-training objective cannot reward fitting technical noise — the quantized target of a masked cell is one of a few hundred stable states, not a point in a continuum that noise can drag around. Second, discrete tokens are a compact intermediate layer that a lightweight downstream model can consume, exactly as discrete speech units let a small sequence-to-sequence student match a much larger model on speech translation (BLEU 9.13 versus 9.00 for full ASR pre-training, with roughly half the parameters) Compact Speech Translation Models via Discrete Speech Units Pretraining. In our setting, the tokens should be things a tumor immunologist can name: an M2-polarized macrophage state, an exhausted CD8+ T cell state, a barrier-forming fibroblast state.
Following wav2vec 2.0 wav2vec 2.0, the pre-training objective for one masked cell is
$$\mathcal{L}_{\text{contrast}} = -\log \frac{\exp\big(\text{sim}(c_i, q_i)/\kappa\big)}{\sum_{\tilde{q} \in \{q_i\} \cup D_i} \exp\big(\text{sim}(c_i, \tilde{q})/\kappa\big)},$$
where $c_i$ is the context embedding the encoder produces for masked position $i$ in a tissue sequence, $q_i$ is the true quantized state token of the cell that was masked, $D_i$ are quantized states of distractor cells sampled from the same tissue region (so the model must distinguish genuinely different states, not different regions), $\kappa$ is a temperature, and a codebook-diversity term is added to prevent all cells collapsing onto a few tokens.
Research Program
We build three pieces. The tokenizer maps a cell's expression profile to a token via product quantization with a Gumbel-softmax relaxation (differentiable assignment to a learned codebook), trained jointly with the encoder; we will compare a flat codebook against the residual, hierarchical codebooks that the discrete-speech-units literature flags as necessary when a fixed $K$-means vocabulary fails to span the phenomenon space Compact Speech Translation Models. The pre-trained encoder is a Transformer that reads cells in spatial order — cells within a neighborhood become a sequence, the direct analog of audio frames — and is trained on the contrastive objective above, using unlabeled data at speech scale: scTab's 22.2 million human single-cell profiles (the corpus scJEPA pre-trains on scJEPA) plus any unlabeled tumor atlases we can attach without labels. The decoupled adapter is a small sequence model, trained only on the labeled clinical cohort, that reads token sequences and predicts the clinical endpoint — responder versus non-responder, or a continuous barrier score. The decoupling is borrowed deliberately: the DSU pipeline showed that putting the heavy unlabeled learning in an unsupervised stage and the scarce supervision in a tiny second stage beats end-to-end training under label scarcity Compact Speech Translation Models.
Interpretability is a first-class deliverable, not an afterthought. Because tokens are discrete, we can audit them against known immunosuppressive biology: does one codebook entry capture the CCL2-driven macrophage recruitment state that MYBL2 induces in ovarian cancer The MYBL2–CCL2 axis, and another the MTA-accumulation-driven M2 polarization of MTAP-loss glioblastoma Landon J. Hansen et al., and another the exhausted CD8+ state described in single-cell checkpoint-inhibitor atlases Dissecting the tumor microenvironment in response to immune checkpoint inhibitors via single-cell and spatial transcriptomics? If the codebook rediscovers these published states without ever seeing their labels, that is strong evidence the tokens are biological, not artifacts.
Baselines
The naive reference point is unsupervised $K$-means clustering of cell profiles (cluster proportions plus a logistic regression on patients), with no pre-training and no learned encoder — the "do nothing" floor that any method must clear to justify its complexity. The strongest existing competitors are the continuous-latent self-supervised models: scJEPA scJEPA and mask-sc Masked Modeling for Single-cell Clustering of scRNA‐seq Data, both run by us on the identical pre-training corpus (scTab), identical downstream cohorts, and an identical tuning budget — same hyperparameter search count and same GPU-hours — so the comparison isolates the quantization ingredient rather than compute. On 10x Genomics PBMC cells, mask-sc reports an ARI of 80.64, NMI of 79.23, and silhouette of 69.41, a gain of 12.01 ARI over contrastive-sc Masked Modeling for Single-cell Clustering of scRNA‐seq Data; on the same PBMC task, Cell-JEPA reports a 36% relative improvement in AvgBIO over scGPT zero-shot, though its exact scalar values have no reported result found on Lacuna. For the spatial tasks, the supervised reference is the ColorAE + U-Net ensemble, which reaches an F1 of 0.99 on detecting K17-negative tumor cells in the 23-slide pancreatic cohort A Novel Framework for Characterization of Tumor-Immune Spatial Relationships; for response prediction, the external yardstick is BFReg-NN's C-index of 0.90 for drug-synergy prediction in triple-negative breast cancer AI-Powered Omics-Based Drug Pair Discovery for Pyroptosis Therapy Targeting Triple-Negative Breast Cancer, noting honestly that it was trained on a different cancer type and cohort, so it sets an ambition level rather than a same-data rival. The decisive ablation is our own architecture with the codebook removed: the same Transformer, the same masked positions, but a continuous latent target (JEPA-style) instead of a quantized token. If the discrete version does not beat that ablation at small labeled budgets, the core claim is wrong and we will say so.
Why not just…?
Why not just deconvolve bulk RNA-seq with CIBERSORT or TIMER? Because bulk deconvolution is documented to fail at exactly the distinctions this proposal targets. It suffers statistical co-linearity that makes closely related immune cell types indistinguishable, a "purity paradox" that confounds tumor content with immune signal, and "molecular mimicry," in which tumor cells express immune markers like CTLA-4 that the algorithms attribute to infiltrating immune cells Dvir Aran, Atul J. Butte et al.. More fundamentally, bulk data averages away the spatial architecture — the fibroblast barrier score Spatially resolved C1QC+ macrophage-CD4+ T cell niche and the roughly 100µm influence zones of K17+ tumor nests A Novel Framework for Characterization — which is the physical mechanism of immunosuppression itself. Bulk deconvolution cannot even represent the thing we want to predict.
Why not just train a supervised network on the annotated slides and cohorts? The annotation bottleneck is identical to the transcription bottleneck that motivated self-supervised speech: 23 expert-annotated slides is not enough to train a deep model, and supervised U-Net pipelines demand exhaustive pathologist pixel labels A Novel Framework for Characterization. Where supervision exists, it is near-saturated on the easy task — the supervised ensemble already hits F1 0.99 on K17-negative cell detection — but the open task is spatial influence and clinical response, and clinical response cohorts are small with response rates of only 6% to 22% in ovarian cancer The MYBL2–CCL2 axis. Supervised-only models on cohorts of that size overfit; the label-efficiency experiment below tests this directly by comparing supervised-only adapters to pre-trained ones at 10, 20, and 50 labeled patients.
Why not just use scJEPA, Cell-JEPA, or mask-sc as they already exist? This is the real threat, and we do not argue around it — we run them as our primary baselines under equal compute. But the literature itself predicts where they should crack. mask-sc degrades on small cohorts (it failed on Quake Smart-seq2 Limb Muscle) and suffers pretext overfitting on complex human tissues Masked Modeling for Single-cell Clustering; continuous masked reconstruction fits technical dropout artifacts Cell-JEPA; and integration-style training overcorrects, erasing precisely the fine-grained intra-cell-type transitions — M1-to-M2 macrophage polarization, gradations of T-cell exhaustion — that constitute immunosuppression Benchmarking deep learning methods for biologically conserved single-cell integration. If, at matched budget, the continuous baselines match our discrete model on the tumor-specific label-efficiency curves, the honest conclusion is that quantization adds nothing here, and the proposal's contribution reduces to a negative result worth reporting.
Proposed Experiments
First, zero-shot and few-shot cell-state recovery. Pre-train on scTab, then evaluate clustering on PBMC-10K (directly comparable to mask-sc's reported ARI of 80.64 on 10x PBMC Masked Modeling for Single-cell Clustering) and on tumor single-cell atlases, where no reported result exists on Lacuna for the continuous baselines and we will generate them ourselves. The tumor evaluations are the ones that matter, because PBMC is blood, not a microenvironment.
Second, the label-efficiency curve, the speech benchmark's signature experiment transplanted: wav2vec 2.0 showed that pre-training plus 10 minutes of labels beats prior systems trained on 100 hours wav2vec 2.0. On the colorectal cancer spatial atlas (314,774 cells across 50 imaging-mass-cytometry patients, 50 spatial proteomics patients, and 9 spatial transcriptomics patients Spatially resolved C1QC+ macrophage-CD4+ T cell niche), we sweep labeled patients from 10 upward and predict responder versus non-responder and the continuous barrier score, measuring AUC against the clinical labels and correlation of predicted barrier scores with progression-free survival. The biological sanity check is the published composition gap: C1QC+ resident tissue macrophages comprise 31.8% of the macrophage pool in complete responders versus 13.6% in non-responders Spatially resolved C1QC+ macrophage-CD4+ T cell niche — our token proportions should recover this separation without supervision.
Third, spatial influence without full annotation. On the 23-slide pancreatic mIHC cohort, we pre-train the tokenizer on unlabeled tiles and fine-tune the adapter to predict the spatial deviation statistics of CD8+ and CD163+ cells around K17+ nests (published mean absolute deviations of 1.36 and 0.99 respectively A Novel Framework for Characterization) using a fraction of the pathologist tiles the supervised pipeline needed, comparing against the supervised U-Net baseline at matched annotation budget. Fourth, clinical response with token adapters on the 129-patient ovarian cohort The MYBL2–CCL2 axis and the 360-patient FUSCC triple-negative breast cancer cohort AI-Powered Omics-Based Drug Pair Discovery, reporting concordance against the same-cohort supervised baselines and against BFReg-NN's C-index of 0.90 as an external ambition level.
The decisive checks are threefold: the quantization ablation must show the discrete model beating its own continuous-target twin specifically at small labeled budgets; the codebook audit must show named immunosuppressive states (CCL2-recruited macrophages, MTA-polarized macrophages, exhausted CD8+ T cells) emerging as individual tokens; and the label-efficiency curve must cross below the supervised-only baseline somewhere on the 10-to-50-patient sweep. Any one of these failing falsifies the core claim in a way we can state precisely.
References
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- A Novel Framework for Characterization of Tumor-Immune Spatial Relationships in Tumor Microenvironment
- Spatially resolved C1QC+ macrophage-CD4+ T cell niche in colorectal cancer microenvironment: implications for immunotherapy response
- The MYBL2–CCL2 axis promotes tumor progression and resistance to anti-PD-1 therapy in ovarian cancer by inducing immunosuppressive macrophages
- Cell-JEPA: Latent Representation Learning for Single-Cell Transcriptomics
- Masked Modeling for Single-cell Clustering of scRNA‐seq Data
- scJEPA: Dual-Space Self-Supervised Learning for Single-Cell Representations
- Compact Speech Translation Models via Discrete Speech Units Pretraining
- MTAP loss correlates with an immunosuppressive profile in GBM and its substrate MTA stimulates alternative macrophage polarization
- Dissecting the tumor microenvironment in response to immune checkpoint inhibitors via single-cell and spatial transcriptomics
- AI-powered omics-based drug pair discovery for pyroptosis therapy targeting triple-negative breast cancer
- Digitally deconvolving the tumor microenvironment
- Benchmarking deep learning methods for biologically conserved single-cell integration