Separating Clone-Inherited from Niche-Induced Gene Expression in Tumor Tissue

When a tumor region switches on a gene, two explanations compete. The cells may belong to a subclone (a genetically distinct branch of the tumor's family tree) that carries the program in its DNA. Or the local tissue neighborhood, such as contact with stroma, hypoxia near necrosis, or immune infiltration, may be inducing it. Spatial transcriptomics measures gene expression at known tissue locations, but it cannot tell these apart. Each measured spot is a mixture of cells, and clones grow in contiguous patches, so "which clone" and "where in the tissue" are almost perfectly confounded. We propose to separate them by treating the clonal tree as a covariance structure. Each spatial cell or spot is placed onto a fixed reference clonal tree built from single-cell data. The niche is described by multi-scale histology features that recur across clone territories, rather than by raw coordinates. Each gene's expression variance is then split into a part shared along tree branches and a part explained by morphological niche.

In two sentences: we borrow phylogenetic placement to put every spatial measurement on a tumor's family tree, and we borrow multi-scale pathology features to describe each location's micro-environment. A per-gene mixed model on mixed-cell spots then reports what fraction of differential expression is inherited and what fraction is induced. The tree supplies the inheritance structure. Histology supplies a niche description that is not collinear with clone territory. Count-level mixture modeling handles the fact that spots blend many cells.

Core Claim

Differential expression in spatially profiled tumors can be decomposed, gene by gene, into a lineage component and a niche component. This becomes identifiable when the niche is defined by morphology that repeats across clones, instead of by position, which tracks clone. The lineage component should be structured by the clonal tree rather than by flat clone labels.

The closest existing work handles one confounder at a time. POLARIS models spot counts as Negative Binomial mixtures of reference cell types and tests layer-specific expression while controlling for composition. It treats cell types as static categories and has no lineage. 2DE removes false niche markers caused by transcripts bleeding in from neighbouring cells, but it assumes predetermined phenotype classes. PhylEx reconstructs clonal trees from single-cell RNA and bulk DNA and finds clone-specific programs such as an immunoediting signature. It works on dissociated cells, so it cannot ask whether a program is clone-intrinsic or niche-driven. Those three papers motivate the gap. DeepSpot2Cell and TRIPLEX supply the method for niche description: multi-scale image encoders at the cell, spot and neighbourhood scales. DeepSpot2Cell also supplies the additive spot-equals-sum-of-cells assumption we reuse. Neither model has any lineage notion.

Model

Take a spatial unit $s$ (a 55µm spot, or a segmented cell) containing cells $i \in s$. Each cell has a reference cell type $t_i$, a placement $b_i$ on a branch of the reference clonal tree, and a multi-scale histology embedding $h_i$. For tumor cells and gene $g$:

$$\log \mu_{ig} = \beta_{g,t_i} + u_g(b_i) + v_g(h_i), \qquad y_{sg} \sim \mathrm{NB}\Big(\textstyle\sum_{i\in s}\ell_s\,\mu_{ig},\ \phi_g\Big).$$

Here $\beta_{g,t}$ is the baseline for that cell type. The lineage effect $u_g$ is a Gaussian process whose covariance between two placements is the shared root-to-ancestor branch length on the clonal tree. This is the Brownian-motion covariance used in comparative phylogenetics, so effects acquired on a branch are inherited by all its descendants. The niche effect $v_g$ is a Gaussian process with a kernel on histology embeddings. $\ell_s$ is library size and $\phi_g$ is a dispersion, shrunk across genes as in edgeR and DESeq2. The quantity we report per gene is $\lambda_g = \sigma^2_{u,g}/(\sigma^2_{u,g}+\sigma^2_{v,g})$, the inherited fraction, together with branch-level effect estimates.

Placement works as in pplacer. The reference tree is built once, with PhylEx, from matched single-cell RNA and bulk DNA. Each spatial unit's likelihood is then evaluated on every branch, which is linear in the number of branches. Evidence comes from allelic counts at the tree's mutations where coverage permits, and from expression-inferred copy-number profiles. The tree's covariance enters the model as an expectation over the placement posterior, so poorly placed spots contribute weak lineage information rather than wrong information.

Keeping the topology fixed is deliberate. Collienne et al. found that adding a taxon changed the tree in 88.62% of cases, often far from the insertion point, so we never let spatial data rewrite the tree. Instead we follow PhyloTune, which pairs placement with novelty detection. A spot is flagged as a possible unseen clone when its best-branch likelihood falls below a calibrated threshold. In that case the reference tree is rebuilt, not patched. This also makes new sections or longitudinal biopsies cheap to analyse without rerunning global MCMC.

Research Program

The work has three parts.

First, we build the placement module and calibrate it. On held-out single cells whose clone is known from PhylEx, we measure how placement accuracy degrades as we move from full-transcriptome to targeted-panel gene coverage and from single cells to multi-cell spots.

Second, we build the variance-decomposition model. Inference is variational, with the cell-level latent sum handled as in DeepSpot2Cell's additive aggregation. Histology embeddings come from pretrained pathology foundation models at three scales. Separately, we analyse identifiability. We expect $\lambda_g$ to be recoverable only when niche states recur across at least two clone territories. We will state this as a measurable condition (the overlap of the histology-niche distribution across clones) and report it for every dataset.

Third, we construct a semi-synthetic benchmark where the truth is known. We start from the 10X Xenium lung, breast and pancreas sections used by DeepSpot2Cell. Tumor cells are assigned to clones on a simulated tree by spatially contiguous growth. Lineage effects are injected along branches, and niche effects are injected as functions of real morphology, for example distance to stroma as segmented from H&E. Cells are then aggregated into simulated Visium-style spots, with transcript bleed simulated as in 2DE's setting.

Baselines

The naive reference is the standard two-step pipeline. Spots are deconvolved with Cell2location or CARD, which Li et al. identify among the top performers across 50 datasets. Spots are clustered into clones by inferred copy number. DESeq2 is then run with clone and spatial-domain covariates. It is fair because it uses the same deconvolution reference, clone calls derived from the same tree, and the same gene set. It is beaten if our method recovers ground-truth lineage-versus-niche labels with higher AUROC and lower false attribution. We expect this baseline to attribute niche effects to whichever clone occupies the niche.

The strongest existing competitor is POLARIS run with clone territories as its "layers". This is exactly its composition-controlled location test, and it is the most direct existing way to ask the question. No reported result was found on Lacuna for POLARIS on a lineage-versus-niche task. We will therefore run it on our benchmark with its published defaults and with the same tuning budget we give ourselves: a fixed grid over regularisation and dispersion settings, chosen on a held-out section.

For the contamination axis we compare against 2DE. Its reported benchmark is precision on Xenium breast cancer niche markers with Proseg segmentation, which rose from 0.47 to 0.88 when 2DE filtered t-test results. We will evaluate on that same benchmark and treat 0.88 as the bar our niche component must meet or exceed. That check tells us our model does not lose 2DE's gain while adding lineage attribution. For clone recovery itself, PhylEx's V-measure of 0.87 on HGSOC (versus about 0.49 for bulk-first methods) is the reference against which placement-derived clone assignments of held-out cells are compared.

Three ablations isolate the key ingredients. The first replaces the tree covariance with a star tree (flat clone labels). The second replaces the histology niche kernel with a kernel on xy coordinates. The third uses hard maximum-likelihood placement instead of the placement posterior.

Why not just…?

Why not just use flat clone labels as a covariate in DESeq2? This is the most serious threat, and the star-tree ablation settles it directly. A tree should help when clones are nested and some clones are small. Sharing along branches lets a minor subclone borrow strength from its parent, and branch-level attribution says where in evolution a program appeared. If the star tree matches the full tree on the semi-synthetic benchmark across clone-size regimes, we will report that the tree adds nothing for estimation. We would then keep it only as the placement scaffold.

Why not just model the niche with a spatial Gaussian process on coordinates? Because clones occupy contiguous patches, a smooth function of position can absorb almost any clone effect. The decomposition then collapses. The xy-kernel ablation tests this. We predict it will drive $\lambda_g$ toward zero, or make it unstable, whenever clone territory and position coincide. Histology niches avoid this because, for example, a stromal interface appears inside several clone territories. Where niche states do not recur across clones, we expect our method to fail as well and will say so via the overlap diagnostic.

Why not just concatenate multi-scale image features into the niche covariate? Shrivastava et al. found that flat multi-zoom concatenation did not beat single-scale models. We therefore compare single-scale, concatenated and hierarchically fused embeddings (TRIPLEX-style) within the niche kernel. We adopt the simplest one that matches the best on niche-effect recovery. Multi-scale fusion is not the contribution here, and we will not defend it if a single scale suffices.

Why not rebuild the full clonal tree with the spatial data included? Restarting global MCMC for every new section is the cost that placement avoids. PhyloTune reports 4 minutes versus over 20 hours for a comparable shift from global to local updating. Spatial spots are mixtures with very sparse allelic signal, so including them in tree search would add noise to the topology. The instability results of Collienne et al. argue for freezing the topology and routing novelty to an explicit rebuild.

Decisive Checks

The first decisive check is on the semi-synthetic Xenium benchmark. Across a sweep of effect sizes and of niche-recurrence across clones, the full model must estimate $\lambda_g$ with lower error than all baselines and ablations. It must also keep attribution AUROC above chance in the regime where the xy-kernel variant collapses.

The second check uses real tumor sections with matched single-cell references. There is a natural positive control for the lineage component: genes inside copy-number-amplified regions specific to a branch should receive high $\lambda_g$ on that branch through dosage alone. Canonical stromal-contact or hypoxia-response genes should receive low $\lambda_g$. A method that cannot separate these two control sets fails, regardless of simulation performance.

The third check is on the 2DE Xenium breast benchmark. Precision of niche markers from our niche component must match 2DE's reported 0.88.

Finally, we place held-out sections without rebuilding the tree. Placement-derived clone maps must agree with maps from a full rebuild, and the novelty flag must fire on sections from which a known clone was withheld from the reference.