StromaBench: A Living Benchmark for Predicting Microenvironment Remodeling in Solid Tumors

In many solid tumors, immunotherapy fails not because immune cells cannot kill, but because they cannot get in. A stiff extracellular matrix — dense cross-linked collagen and hyaluronic acid — walls off the tumor core, so cytotoxic T cells and drugs stall at the periphery. A growing toolkit can dismantle that wall: engineered NK and T cells that secrete matrix-digesting enzymes (CAR-NK cells for cancer immunotherapy: recent advances and future directions), decorin-armed oncolytic viruses (Decorin-armed oncolytic adenovirus promotes natural killers (NKs) activation and infiltration to enhance NK therapy in CRC model), LOX inhibitors and TGF-β traps that soften the matrix (Modulating extracellular matrix stiffness: a strategic approach to boost cancer immunotherapy), and enzyme-responsive nanoparticles that in mice converted minimal survival into 60% survival at day 18 (Enzymatically responsive nanocarriers targeting PD-1 and TGF-β pathways reverse immunotherapeutic resistance and elicit robust therapeutic efficacy). Yet there is no computational way to answer the question a clinician actually faces: given this patient's pretreatment tissue, which remodeling intervention will open this particular tumor? Phase III failures of TGF-β/PD-L1 agents trace partly to exactly this inability to pre-identify responders (Bispecific antibody targeting TGF-β and PD-L1 for synergistic cancer immunotherapy).

The reason is that no benchmark asks this question. The living perturbation benchmark OP3 predicts how dissociated blood cells respond to chemical compounds (A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types) — liquid cells with no matrix, no spatial coordinates, no immune-exclusion geometry. The living deconvolution platform DECONbench estimates cell proportions in simulated, untreated mixtures (DECONbench: a benchmarking platform dedicated to deconvolution methods for tumor heterogeneity quantification). The large spatial benchmarks, HESCAPE and the 18-method spatial deconvolution study, evaluate frozen snapshots of tissue that never changes under therapy (A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics; A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics). The proposal here is to build the first benchmark — living, rotating, and code-submission-based — whose prediction target is the remodeling trajectory itself.

Core Claim

The claim is that predicting response to stroma-remodeling therapy is a well-posed learning problem that current benchmarks structurally cannot measure, and that a benchmark organized around the spatial transition under remodeling interventions will expose which model ingredients matter. Concretely, the task is: given a pretreatment paired profile of a tumor — whole-slide H&E plus spatially resolved transcriptomics — and a candidate remodeling intervention, predict the post-treatment spatial state. The primary endpoint is immune infiltration depth, the distance cytotoxic lymphocytes penetrate past the tumor–stroma boundary:

$$\mathcal{E}_{\text{depth}} = \frac{1}{|C|}\sum_{c \in C}\big|\, d_{\text{pred}}(c) - d_{\text{obs}}(c) \,\big|$$

where $C$ is the set of tumor nests in a slide and $d(c)$ is the observed or predicted 90th-percentile distance of CD8+ T or NK cells from the stromal margin of nest $c$. Secondary endpoints are change in stromal density, cell-state shifts (fibroblast-to-myofibroblast transition, macrophage polarization), and proportion error in mean absolute error — directly comparable to DECONbench's reported floor of 0.024 (DECONbench) — plus a clinically framed stratification endpoint. What makes this a dynamic benchmark rather than a new dataset is the evaluation regime: hidden test slices that rotate quarterly on a LiveBench-style cadence (LiveBench: A Challenging, Contamination-Limited LLM Benchmark), submissions as Docker-contained source code executed server-side against data the submitter never sees, following DECONbench's defense against the self-assessment trap, and mandatory variance reporting, which 14 of 24 major benchmarks currently omit (BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices).

Research Program

The program has three phases. Phase one seeds the platform with simulated remodeling trajectories by extending DECONbench's Dirichlet mixture generator with explicit transition dynamics: starting from its PDAC mixtures, we parameterize post-treatment states using measured murine shifts from the gelatinase-responsive nanoparticle study — suppressed fibroblast-to-myofibroblast transition, macrophage polarization away from M2, heightened CD8+ infiltration across its 10,726 profiled cells (Enzymatically responsive nanocarriers...). This gives an immediately available hidden test set whose ground truth is exact, and lets us measure how much the best static deconvolution methods degrade when the mixture they must unmix has moved — a question DECONbench's authors flagged but could not test. Phase two replaces simulation with reality: we ingest longitudinal murine remodeling cohorts and pretreatment/post-treatment human Xenium-plus-H&E pairs in the HESCAPE format (620,000 image–gene pairs across 54 donors demonstrates the data volume exists (HESCAPE)), reserving a rotating fraction as hidden evaluation. Phase three is the evaluation science: multi-metric reporting with bootstrap intervals, ontology-informed error metrics borrowed from the single-cell foundation model benchmark (Biology-driven insights into the power of single-cell foundation models), and release-date watermarking of hidden slices so that any model whose training predates a slice's public date can be scored for contamination, the LiveBench method.

Two design commitments follow from evidence in the file rather than taste. First, because the benchmark-lottery meta-analysis showed that aggregate scores over 70 task subsets crowned 6 different winners (The Benchmark Lottery), the leaderboard reports per-endpoint results and rank stability across refreshes, not one scalar. Second, because HESCAPE found that contrastive cross-modal pretraining degraded direct gene-expression prediction relative to an un-pretrained Gigapath encoder, the benchmark treats foundation-model superiority as a hypothesis to test, not a premise.

Baselines

The do-nothing reference is the identity predictor: the post-treatment state equals the pretreatment state, with a population-mean-trajectory variant that predicts the cohort-average change. This is a serious floor, not a straw man — OP3 found that submitted models predict quiescent (unchanged) states far better than significant differential expression (OP3), so on tumors that barely remodel, doing nothing scores well. The identity baseline tells us how much headroom any learned model actually has, and we report all results separately on the significant-remodeling subset where the task lives.

The strongest existing method differs by endpoint. For proportion estimation it is DECONbench's top method, the NMF-based r_WNM, which reported a mean MAE of 0.024 on static PDAC mixtures (DECONbench); we run it out-of-the-box on remodeled mixtures and tissue, and the benchmark's first headline measurement is by how much that 0.024 inflates under remodeling shift. For the clinical stratification endpoint it is the VGG-19 Deep-immune score pipeline of Yang et al., which on 1,010 colorectal cancer patients separated 5-year survival at 87.4% versus 58.2% (hazard ratio 0.27, 95% CI 0.15–0.48) and improved the integrated AUC from 0.713 to 0.726 when added to clinical variables (Artificial intelligence for quantifying immune infiltrates interacting with stroma in colorectal cancer); beating it means exceeding an integrated AUC of 0.726 on pretreatment-only inputs for predicting post-treatment infiltration, with non-overlapping bootstrap intervals. For the perturbation track we reimplement a representative OP3-style winner — a transformer/LSTM ensemble with pseudolabeling — under an equal tuning budget; no reported result found on Lacuna for the OP3 competition winners' MRRMSE, so we cannot quote a number and must reproduce one ourselves under matched compute. Every baseline, including ours, runs through the same Docker execution, the same hidden slices, and the same fixed hyperparameter-search budget.

The key ablation is a spatially ablated twin: the identical model trained on the same data with spatial coordinates removed or randomly permuted, emulating dissociated profiling. If the full model does not clearly beat its ablated twin on $\mathcal{E}_{\text{depth}}$, the central claim — that spatial context is the missing ingredient in perturbation prediction — fails. A second ablation, in the cross-modal track, pits un-pretrained Gigapath against contrastively pretrained encoders, since HESCAPE found the un-pretrained baseline achieved the highest Pearson correlation on direct expression prediction (HESCAPE). And following the finding that scVI and logistic regression remained highly competitive with six single-cell foundation models (Biology-driven insights...), simple classical baselines are mandatory entries; if they win, that is a legitimate and publishable outcome of the benchmark.

Why not just…?

Why not just build a static benchmark with hidden test splits? Hidden splits do capture most of the contamination and self-assessment benefit, and if cohort accrual stalls the living machinery adds nothing — this is a real weakness, not one to argue around. The evidence for rotation is that static aggregates mislead: 6 different top-1 winners across 70 task subsets (The Benchmark Lottery), and 17 of 24 benchmarks lacking reproducible scripts (BetterBench). But the honest test is empirical, and we build it in: we measure Kendall rank correlation of the leaderboard across consecutive refresh snapshots, echoing the ~0.49 single-game correlation the lottery paper found in Atari. If rankings are stable across refreshes, the dynamics claim fails and the benchmark should collapse to a well-maintained static suite — we will say so.

Why not just extend OP3 with stroma data? OP3's plumbing — cloud execution, code submission, a compound-by-cell-type-by-gene tensor — is exactly what we reuse, so the infrastructure objection is granted in advance. The science is what cannot be retrofitted: OP3's data are dissociated peripheral blood cells, so the signal this benchmark scores — where lymphocytes sit relative to stromal boundaries, and how far they penetrate after the matrix is softened — does not exist in that tensor. The relevant matrix biology (LOX cross-linking, interstitial pressure, CAF activation) is precisely what dissociation destroys (Modulating extracellular matrix stiffness...). The spatial-ablation experiment is designed to settle this directly: if a model trained on OP3-style dissociated inputs matches the spatially trained one, the objection wins and the field should simply extend OP3.

Why not just use existing H&E scoring, like the Deep-immune score? This is the strongest objection because Yang et al.'s pipeline is cheap, clinical, and validated against immunohistochemistry with 5-year survival separation and an integrated AUC of 0.726 (Yang et al.). But it is a prognostic score on retrospective, untreated cohorts — it stratifies outcome, it does not predict response to a specific remodeling intervention, which is the selection problem that sank Phase III TGF-β trials (Bispecific antibody targeting TGF-β and PD-L1...). It also operates on 224×224 patches and its own authors note it cannot resolve fiber-level matrix change or the molecular state shifts (M2-to-M1 repolarization, CD38+/CD69+ activation) that scRNA-seq and cyTOF measured in the remodeling mouse studies (Enzymatically responsive nanocarriers...). We include it as a baseline rather than dismiss it: if pretreatment Deep-immune scoring already predicts post-treatment infiltration as well as multimodal models, the experiments will reveal that, and it would change what this benchmark recommends to the field.

Why not just accept that simple models win and skip the benchmark? Perhaps, but this is unfalsifiable without the benchmark. The precedent cuts both ways: simple averaging matched complex multi-omic integration in DECONbench (a 2.12% gain for b_COM), yet un-pretrained and pretrained vision encoders split outcomes by task in HESCAPE. Only a shared, hidden, rotating evaluation can say which regime holds when the perturbation is a physical intervention rather than a compound.

Proposed Experiments

The first experiment is the degradation study: run all eight DECONbench deconvolution methods on simulated remodeled mixtures and quantify how far the 0.024 MAE floor rises under therapy-induced state transition (DECONbench). The decisive check is whether the inflation is at least twofold on the significant-remodeling subset; if static methods are robust to the shift, the motivation weakens and we report it. The second experiment seeds real evaluation on longitudinal murine remodeling cohorts in the gelatinase-responsive nanoparticle format (n=6 per group, 10,726 profiled cells as the pilot scale, GPNP study) and on pretreatment/post-treatment human Xenium-H&E pairs, with $\mathcal{E}_{\text{depth}}$, proportion MAE, and stratification AUC as endpoints. The third experiment runs the rotation machinery: roughly one-sixth of hidden slices refreshed quarterly, each watermarked with its earliest possible training date, following LiveBench's cadence (LiveBench); the decisive check is the rank-stability correlation across snapshots and the measured performance gap between watermarked and pre-watermark slices. The fourth experiment tests the cross-modal paradox under remodeling: whether contrastive pretraining helps or hurts prediction of post-treatment spatial expression from pretreatment H&E, against the un-pretrained Gigapath baseline that HESCAPE showed was strongest on direct expression prediction (HESCAPE).

The proposal succeeds if three things hold at the end of year two: some model beats the identity baseline on the significant-remodeling subset, beats the Deep-immune pipeline's 0.726 integrated AUC on pretreatment-only prediction of post-treatment infiltration, and the spatial ablation of that model collapses its advantage. If none hold, the benchmark will still have produced the first quantitative evidence that remodeling prediction is either solvable with today's tools or genuinely open — and either answer moves a field that currently runs every decision on n=6 mice.