Speech synthesis ran into a specific failure: the same sentence can be spoken with many valid melodies, so a model trained to predict the melody with squared error learns their average. The result is a flat, robotic voice. The field's fix was to cut each continuous prosodic quantity, such as pitch range, pitch slope, duration or loudness, into small numbered bins and predict a probability distribution over those bins with a language model. ProsodyLM does this with five word-level prosody features. After pretraining on 29.9k hours of unlabeled audiobooks, its model shows abilities it was never trained for, such as placing emphasis and recognizing emotion. We argue that tumor tissue presents the same statistical problem. Given the cells around a T cell, that T cell may plausibly be exhausted or active. The mix of neighbors does not settle it, and those two possibilities are separate modes, not points on a single continuum. Any model that regresses a cell's protein levels from its neighborhood will blur those modes together. That blurring hides the variation that matters most for whether a checkpoint-inhibitor drug works.
We propose to pretrain a transformer on very large numbers of unlabeled cells from multiplexed imaging (images that measure dozens of proteins in every cell while keeping each cell's position). Each cell is written as quantized tokens for marker level and for the distance to each neighbor. The model learns by predicting masked cells' tokens from their spatial context. We then predict immunotherapy response with a frozen model and a tiny linear readout. The reason for this design is practical. Response labels exist for about a dozen patients per cohort, while unlabeled cells number in the millions. The bet is that the label-hungry part of the problem can be moved onto the cells.
Core Claim
Our claim is that discrete, distribution-valued prediction of masked cell states produces neighborhood representations that separate responders from non-responders better than three alternatives: continuous regression with the same architecture, the rank-ordered "cell sentences" used by prior work, and hand-engineered niche statistics. The comparison is made at a fixed and very small labeled-patient budget. The mechanism is the one the prosody literature identified. The squared-error optimum for a masked cell's marker vector $x$ given its neighborhood $c$ is the conditional mean $\mathbb{E}[x \mid c]$. When $p(x \mid c)$ is bimodal, as with exhausted ($TIM3^+$, $TIGIT^+$) versus effector ($CD57^+$, $OX\text{-}40^+$) states, that mean lies between the modes and corresponds to no real cell. A categorical output over binned values keeps both modes, and keeps how much probability each one gets. We expect this mode balance to carry the resistance signal. In the AML CODEX cohort of Park et al., non-responders kept exhausted phenotypes in myeloid-dominated niches while responders shifted toward effector states.
Relation to Prior Work
The closest work is Spatial2Sentence. It turns imaging mass cytometry cells into "sentences" by listing protein names in order of expression, groups spatial neighbors into contrastive prompts for a pretrained language model, and reports gains of 5.98% in cell typing and 4.18% in clinical-status prediction on a diabetes dataset. It differs from our proposal in three ways. Ranking throws away magnitudes and distances. It borrows a text-pretrained model instead of pretraining on cells. It was never tested on immunotherapy response. We keep its core move of treating a neighborhood as a context in which a cell is predicted. We replace ranks with quantized magnitudes and distances, following ProsodyLM, and we treat the rank encoding as a baseline instead of an assumption.
The response-prediction side supplies both the motivation and the bar to beat. Trevino et al. found that in resistant hepatocellular carcinoma, lymphoid aggregates are walled off behind a fibroblast- and SPP1-rich "interface niche". Their graph neural network on multiplexed immunofluorescence reached AUROC 0.93. SPACE-GM reached ROC-AUC 0.98 on 13 patients and showed that neighborhood arrangement beats cell counts. Both papers establish that local spatial context carries the signal. Neither pretrains on unlabeled cells, and both report numbers from cohorts so small that the confidence intervals are wide. That fragility motivates our label-efficient design. It also constrains our claims: we cannot honestly claim to "beat 0.98" on 13 patients.
Two more speech papers supply analysis tools. HiMuV-TTS splits prosody into an utterance-level latent and phoneme-level latents. Granero Moya et al. found that pause placement is local and solvable by small models, whereas expressive pitch needs wide context and larger models. We borrow this local-versus-global question in place of an architectural commitment. We ask how far the model's context must extend, from immediate contacts to tissue-scale structures such as the mature tertiary lymphoid structures (organized immune clusters inside the tumor) described by Li et al., before response prediction stops improving. ImmunoDiff also uses a generative model for response, but on CT at 96x96x16 voxels. It shows that generative modeling helps response prediction at organ scale. It says nothing about the cellular scale we target.
Research Program
Each cell becomes a short token sequence. Its own markers are quantile-binned per marker within each cohort, which absorbs differences in staining. Each of its $k$ nearest neighbors is represented by a distance-bin token and a coarse cell-type token. The pretraining objective masks a random subset of cells inside a tissue patch and predicts their marker bins by cross-entropy. Patches are sampled at several radii so that context size can later be varied at probing time. The pretraining corpus pools every multiplexed dataset available to us: the HCC PhenoCycler data, the AML CODEX specimens, the HNSCC cohort, and the two IMC datasets from Spatial2Sentence. Data access to the clinical cohorts is the main logistical risk. If the HCC data are unobtainable, the AML and HNSCC cohorts become the primary response tasks.
The readout stays deliberately weak. We pool the frozen per-cell representations over each patient, either as a mean or as a histogram over a small learned codebook, and fit a ridge logistic regression. All evaluation is leave-one-patient-out. The model is pretrained without the held-out patient's cells, so tissue memorization cannot leak into the score.
Baselines
The naive reference is a logistic regression on per-patient cell-type proportions. SPACE-GM reports that spatial arrangement beats counts, but the file gives no exact count-baseline number for that cohort, so we will measure it ourselves. The strongest knowledge-driven competitor is a logistic regression on a few hand-built niche statistics: the fraction of lymphoid-aggregate cells adjacent to SPP1-high fibroblasts (the interface niche of Trevino et al.), niche diversity counted as in Park et al. (37 niche types in responders versus 19 in non-responders), and the myeloid-niche fraction. These features encode exactly what the field already believes, so they are what the learned representation must beat.
The strongest published learned methods are the spatial graph networks. Trevino et al. report AUROC 0.93 for MIF-GNN on pre-treatment biopsies, and SPACE-GM reports ROC-AUC 0.98 on 13 patients. We will reimplement a graph attention network on the same cell graphs and train it end to end under the same leave-one-patient-out folds, with a hyperparameter budget matched to our readout's. We will report those published figures for context but compare only to our reruns.
The rank-sentence baseline follows Spatial2Sentence's encoding exactly, with neighborhood prompts, and pretrains on the same corpus and compute as our model. This isolates the effect of the encoding. The key ablation keeps our architecture, data and compute but replaces the categorical head with continuous mean-squared-error regression on marker values, a masked autoencoder. Beating it is the test of the core claim.
"Beating" a baseline has a specific meaning here. Patient-level AUROC on roughly a dozen patients has a confidence interval wider than most plausible gains. We therefore require a consistent advantage across every cohort, together with a better label-efficiency curve. Those two together count as success. A single higher point estimate does not.
Why not just…?
Why not just use the hand-engineered interface-niche and diversity features? On a cohort this small, this is the most serious threat. These features may already capture most of the separable signal. Our experiments compare against them directly, and we will also test whether adding the learned representation to these features improves the cross-validated likelihood. If it does not, we will report that the pretraining adds nothing to response prediction. The remaining contribution would then rest on the held-out cell-state results below.
Why not just train a graph network end to end, as Trevino et al. and SPACE-GM did? Those models do reach high reported AUROCs. However, their parameters are fit to about a dozen labels, and nothing in the file shows they transfer across cohorts. Our label-efficiency curve and cross-cohort probe measure exactly this. If the end-to-end graph network matches us when trained on all patients and also when restricted to the fewest-label subsets, our premise is wrong.
Why not just pretrain with continuous regression, which keeps more information than bins? This is the alternative the core claim must defeat, and it is a fair one. The averaging argument predicts that continuous regression will lose specifically on cells whose state is bimodal given their context. We therefore score both pretrained models on held-out masked T cells, separately for cells whose neighborhood sits in mixed exhausted/effector regions. If the gap does not appear there, the prosody analogy fails for tissue.
Why not just use Spatial2Sentence's rank sentences with an off-the-shelf language model? Ranks keep the order of markers within a cell but lose how far apart the levels are and how far away each neighbor is. The interface niche is defined by adjacency at a specific distance. The matched-compute rank baseline answers this directly.
Proposed Experiments and Decisive Checks
The first experiment probes generative quality directly. On held-out patients, we compare negative log-likelihood and calibration of masked-cell marker predictions between the categorical model and the regression model, converting the regression model's output to a likelihood with a fitted Gaussian. We also test whether the categorical model's predicted exhausted-versus-effector probability for T cells tracks the observed phenotype split between responders and non-responders in the AML pre-DLI specimens. The second experiment is response prediction under leave-one-patient-out, reported as AUROC with bootstrap intervals on each response cohort. Alongside it, we draw a label-efficiency curve by fitting the readout on random subsets of 4, 6 and 8 patients. The third experiment is the local-versus-global sweep: we probe with context radii ranging from immediate neighbors up to whole-region patches and locate where response AUROC saturates. Saturation at small radii would agree with the interface-niche account. Continued gains at tissue scale would support the tertiary-lymphoid-structure account of Li et al.. The fourth experiment is a counterfactual check. In resistant HCC tissue, we computationally replace interface fibroblasts with tumor cells and ask whether the model's predicted state for nearby lymphoid cells shifts toward effector. A shift would mean the representation has learned the barrier described in the biology and not just a patient-level shortcut. A null result would warn against reading the representation mechanistically.
The proposal will have failed if, on held-out cells, the categorical head does not beat the continuous head in bimodal neighborhoods, or if the frozen representation adds nothing to the hand-built niche features on response. Either result would be informative, and both are cheap to obtain before any claim about clinical prediction is made.
References
- ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
- 1232 Decoding the immunologic networks of the AML marrow microenvironment using CODEX
- Spatial Coordinates as a Cell Language: A Multi-Sentence Framework for Imaging Mass Cytometry Analysis
- 478 Spatial omics and deep learning define an immunotherapy resistance niche in hepatocellular carcinoma
- 1415 Multimodal spatial signatures of immunotherapy response in hepatocellular carcinoma
- Hierarchical and Multi-Scale Variational Autoencoder for Diverse and Natural Non-Autoregressive Text-to-Speech
- A Comparative Analysis of Pretrained Language Models for Text-to-Speech
- Mature tertiary lymphoid structures evoke intra-tumoral T and B cell responses via progenitor exhausted CD4+ T cells in head and neck cancer
- ImmunoDiff: A Diffusion Model for Immunotherapy Response Prediction in Lung Cancer