Spatial transcriptomics machines such as 10X Visium measure gene expression in small circular "spots" across a tissue slice. Each spot pools the RNA of roughly 8 to 20 cells in dense tumor tissue. Current analysis tools mostly answer one question: how many cells of each type sit in each spot. For cancer immunotherapy that is often the wrong question. Whether a macrophage helps or hinders the tumor depends on its state. The pro-inflammatory $TNFSF10^+$ state and the pro-tumor $SPP1^+$ state predict opposite responses to immune checkpoint drugs. Tumor cells also confuse the count by expressing "immune" genes themselves. We propose to borrow a tool from speech enhancement, the soft time-frequency mask. In that field, a network looks at a noisy recording and outputs, for every time-frequency cell, the fraction of energy that belongs to the target voice. Our transcriptomic version is a network that looks at a spot and its neighbors and outputs, for every gene, the fraction of that gene's reads that came from each cell type. That fraction explicitly includes a patient-specific malignant source for which no reference exists.
The masked reads give a per-spot, per-cell-type expression profile. From it one can score macrophage polarization or T-cell exhaustion inside mixed spots, rather than only counting cell types. The central bet is taken directly from the speech literature. Networks trained on many synthetic mixtures learn to separate talkers they have never heard. In the same way, a mask network trained on synthetic spots built from many patients' tumors should separate reads from a new patient's malignant cells, including the immune-mimicking reads that corrupt reference-based unmixing.
Core Claim
Given a fitted mixture, splitting each gene's count among cell types in proportion to the reference profiles is a statistical identity, not an idea. It is the posterior allocation under a Poisson mixture, and it is the transcriptomic twin of the ideal ratio mask. Re-deriving deconvolution "as a ratio mask" would therefore add nothing, and we do not propose it.
What the reference-ratio allocation cannot do is change a gene's split depending on context. Suppose a spot's CTLA-4 reads come from tumor cells in one patient and from regulatory T cells in another. A fixed signature matrix splits them identically in both. Digitally deconvolving the tumor microenvironment documents this "molecular mimicry" and the related purity paradox as a standing failure of reference unmixing.
Our claim is that a learned soft mask can fix this, while a reference-derived one cannot. Such a mask is predicted from the spot's whole expression vector, its spatial neighborhood and optionally its histology image, and it is trained against true per-type read counts on held-out-patient synthetic spots. The fix shows up most strongly on the readouts that matter clinically, namely within-spot cell-state scores. It shows up less on cell-type proportions, where existing tools are already competitive.
How the Pieces Fit
The speech literature supplies the method. Ideal Ratio Mask Estimation Using Deep Neural Networks for Monaural Speech Segregation in Noisy Reverberant Conditions shows that regressing a soft mask that includes an explicit residual term beats binary masks. That residual, $R$ in $\sqrt{D/(D+R)}$, covers noise and late reverberation, and it plays the role of our unreferenced malignant source. Long Short-Term Memory for Speaker Generalization in Supervised Speech Separation shows that mask estimators generalize to unseen talkers when trained across many of them. Its STOI (an intelligibility score) gain on unseen speakers at −5 dB babble was 14.8% for the LSTM versus 10.8% for a feed-forward network. That improvement grew as training speakers were added, which is the scaling behavior we hope to see with training patients.
The same literature also warns us. Those estimators degrade on unseen non-stationary noise (IRM estimation, reverberant conditions). So generalization to new tumor types cannot be assumed and must be measured.
Listenable Maps for Audio Classifiers supplies a secondary ingredient. Its training objective rewards the masked-in signal and penalizes target evidence left in the masked-out residual. We reuse this as an auxiliary loss: the reads assigned to the malignant residual should carry no immune-state signal when scored by a classifier trained on purified immune cells. Polyphonia uses separator-derived masks to calibrate another model's attention. We do not adopt that use. We do take its stated limitation seriously: priors from separators trained on a fixed set of source categories fail on rare sources. This is why our model carries an open "unknown" source rather than a closed set of cell types.
The tumor microenvironment literature defines the target readouts and the reasons to want them. Tumor-associated macrophage clusters linked to immunotherapy in a pan-cancer census reduces macrophage diversity to two states with opposite niches and opposite response to checkpoint drugs. Microlocalization and clinical significance of stabilin-1+ macrophages shows that only intratumoral macrophages are prognostic, not stromal ones. CONFIRM ties nivolumab response in mesothelioma to B-cell aggregates called tertiary lymphoid structures and to inflammatory chemokines. Each of these is a state-in-place quantity, which proportion maps do not deliver.
MAST-Decon motivates our use of spatial context. It shows that borrowing information from neighboring spots with distance, expression and image kernels helps. It also constrains our design, because it is the obvious way to add spatial context without any learned mask.
Research Program
Synthetic training mixtures. We will build synthetic training spots from annotated tumor single-cell RNA-seq that includes malignant cells, drawing on many patients and cancer types. Each pseudo-spot sums 1 to 30 cells, sampled from a spatial layout model with tumor nests, stroma and immune aggregates. We then apply a learned or empirical "platform transform" that mimics the depth and capture biases separating dissociated single cells from tissue spots. A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics identifies this shift, the "platform effect," as the persistent failure mode.
Mask network. Let $x_{s,g}$ be the observed count of gene $g$ in spot $s$, and let $c_{s,g,k}$ be the true count contributed by cell type $k$, where one of the $K$ types is "malignant/unreferenced." The network $f_\theta$ outputs masks $m_{s,g,k}\in[0,1]$ with $\sum_k m_{s,g,k}=1$ (a softmax over types). Its inputs are spot $s$, its spatial neighbors $\mathcal{N}(s)$ and optional image features $h_s$. We train it with a count-space loss:
$$\mathcal{L}(\theta)=\sum_{s,g,k}\ell\big(m_{s,g,k}\,x_{s,g},\;c_{s,g,k}\big)+\lambda\sum_{s}\big\|\,\phi_{\text{imm}}\big(m_{s,\cdot,\text{mal}}\odot x_{s,\cdot}\big)\big\|_1 .$$
Here $\ell$ is a Poisson deviance, so mistakes on highly expressed genes are penalized proportionately. $\phi_{\text{imm}}$ is a frozen classifier of immune-state programs, and the second term is the masked-out residual penalty adapted from L-MAC, weighted by $\lambda$.
Readouts. The masked profiles $m_{s,\cdot,k}\odot x_{s,\cdot}$ feed state scores for each spot, such as the $SPP1^+$ versus $TNFSF10^+$ macrophage ratio and the exhaustion score of $CD8^+$ T cells. They also feed proportion estimates, obtained by fitting the masked profiles to the references. We will release code for generating the synthetic spots and for splitting data by held-out patient and held-out cancer type.
Baselines
The naive reference point is reference-ratio allocation. Proportions are estimated by RCTD, Cell2location or CARD, and each gene's count is then split by $p_k\mu_{g,k}/\sum_j p_j\mu_{g,j}$, where $p_k$ is the estimated proportion of type $k$ and $\mu_{g,k}$ is that type's reference expression of gene $g$. This is the fixed-signature "ideal ratio mask." It is the most important comparison, because any gain must come from context-dependent allocation and not from the masking formalism. It will be run with the malignant reference built from training patients only, exactly as our model sees them.
The strongest existing methods are Cell2location and CARD, which the 50-dataset benchmark ranks with Tangram as the most accurate and robust tools. That paper reports rankings rather than a single headline accuracy figure. No reported result was found on Lacuna for their state-score recovery or for mimicry-gene allocation, because these tools were not designed for either.
For spatial context we add MAST-Decon. It reports a tertiary lymphoid structure dissection ARI (agreement with annotated regions) of 0.575 versus 0.228 for RCTD on hepatocellular carcinoma Visium (MAST-Decon). This is the one concrete external number we must beat on real tumor tissue.
To keep the comparison fair, every method gets the same reference cells, gene panel, training-patient split and a matched hyperparameter search budget. All model-based baselines also receive the same platform-transformed references. Beating them means two things on held-out patients. First, the masked profiles recover macrophage-state and exhaustion scores with lower error than reference-ratio allocation. Second, proportions are no worse than Cell2location's.
Four ablations isolate our ingredients. A binary mask assigns each gene's reads to the argmax type and tests the value of soft allocation. A context-free variant sees only the spot itself. A variant without the residual penalty drops the L-MAC term. A no-malignant-training variant omits tumor cells from the synthetic mixtures and should reproduce the mimicry failure.
Why not just…?
Why not just use reference-ratio allocation from a good deconvolver? This is the real threat. It is principled, cheap and needs no training. Its weakness is that the split for a gene is fixed by the signatures, so it cannot follow patient-specific tumor expression of immune genes. That weakness is documented qualitatively in Digitally deconvolving the tumor microenvironment. The held-out-patient experiment on mimicry genes settles the question directly. If the learned mask does not beat this baseline there, the proposal reduces to existing practice, and we will report it that way.
Why not just smooth across neighboring spots, as MAST-Decon does? Smoothing improves proportion maps and region recovery, but it still assumes fixed per-type signatures inside each spot. Smoothing therefore cannot tell a tumor-derived CTLA-4 read from a regulatory-T-cell one. We include MAST-Decon both as a baseline and as an optional input to our model. If MAST-Decon proportions combined with ratio allocation match our state recovery, then spatial smoothing, not learning, was the active ingredient.
Why not just give Cell2location a patient-matched malignant reference? When matched single-cell data exist, this may close much of the gap. We will run it as an oracle upper bound for the model-based approach. Most archival Visium cohorts lack matched dissociated data. The practical question is therefore how close our unmatched model gets to this matched oracle.
Why not just use single-cell-resolution platforms? Imaging platforms such as MERFISH, which the benchmark used in simulation, avoid mixing but measure targeted gene panels. A great deal of existing tumor data is spot-based. We use single-cell-resolution data as ground truth for our evaluation, not as a competitor.
Decisive Experiments
Mimicry test. On synthetic spots from held-out patients, we measure the fraction of malignant-derived reads of known mimicry genes (CTLA-4, PD-L1, inflammatory chemokines) that are wrongly assigned to immune types. We compare our model, reference-ratio allocation and the matched-reference oracle.
State-recovery test. We measure the correlation between the true and recovered within-spot $SPP1^+$/$TNFSF10^+$ macrophage ratio, stratified by spot purity. We expect the advantage to concentrate in tumor-dense spots, where mimicry is strongest.
Scaling test. We vary the number of training patients and repeat both tests on held-out cancer types. This mirrors the speaker-generalization curve and tells us whether gains are general or only memorized.
Real tissue. On the hepatocellular carcinoma Visium data we must exceed the 0.575 ARI reported for tertiary lymphoid structure dissection. We will also check that masked macrophage states place $SPP1^+$ signal in T-cell-excluded regions and $TNFSF10^+$ signal in immune-rich regions, as the pan-cancer census predicts. This is a qualitative check, not an accuracy figure.
Clinical probe. Bulk RNA-seq is a single giant spot, so the model can be applied to bulk checkpoint-therapy cohorts. We ask whether mimicry-corrected macrophage ratios improve on the census model's independent-cohort AUC of 0.73 (pan-cancer census), using the same cohorts and the same random-forest protocol.
A negative result on the mimicry test is still informative. It would show that the problem flagged by Aran and Butte is not addressable from the spot's own expression context.
References
- Digitally deconvolving the tumor microenvironment
- Ideal Ratio Mask Estimation Using Deep Neural Networks for Monaural Speech Segregation in Noisy Reverberant Conditions
- Long Short-Term Memory for Speaker Generalization in Supervised Speech Separation
- Listenable Maps for Audio Classifiers
- Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration
- Tumor-associated macrophage clusters linked to immunotherapy in a pan-cancer census
- Microlocalization and clinical significance of stabilin-1+ macrophages in treatment-naïve patients with urothelial carcinoma of the bladder
- Constitutive inflammation and epithelial-mesenchymal transition dictate sensitivity to nivolumab in CONFIRM: a placebo-controlled, randomised phase III trial
- MAST-Decon: Smooth Cell-type Deconvolution Method for Spatial Transcriptomics Data
- A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics