Dozens of recent cancer papers sort patients into "high-risk" and "low-risk" groups using the expression of a handful of metabolism-related genes, such as genes tied to lactate production or to disulfide-stress cell death. They then recommend different drugs for each group. These signatures are trained to predict overall survival in public tumor collections such as The Cancer Genome Atlas (TCGA), and they are usually chosen by screening a hundred or more model combinations. Survival, however, depends on more than tumor biology. It also depends on which hospital treated the patient, how far that patient lived from an oncologist, and how carefully that hospital recorded follow-up. Every TCGA sample carries a code naming the institution that contributed it, and these institutions sit in real places with real catchment populations. This proposal asks a plain question: how much of the "metabolic" survival signal in published signatures is really a signal about where the sample came from? It then asks whether geography can predict how badly a signature will perform at a hospital it has never seen.
The concrete contribution is a validation protocol and an audit. The protocol geocodes each contributing institution, links it to measures of travel burden and deprivation in its catchment area, and holds out whole sites or whole regions during validation. The audit applies that protocol to three published metabolic signatures, re-implemented from their gene lists. A colleague could summarize it in two sentences. Metabolic signatures are validated as if the patients came from nowhere, even though survival is shaped by hospital and geography. We measure how much of each signature's apparent power and external failure is explained by site and place, and we test whether spatial structure lets us predict a new site's performance before deploying there.
Core Claim
We claim that survival-trained metabolic signatures partly absorb site-level variation in outcomes. If so, three things should be observable. First, the risk score should predict the contributing institution better than a random gene set of the same size does. Second, a signature derived with whole sites held out should lose more of its discrimination than one validated on random patient splits. Third, the residual site-level survival offset, meaning how much better or worse a site's patients do than the signature predicts, should be spatially clustered. "Spatially clustered" here means positive spatial autocorrelation: nearby sites resemble each other more than distant ones.
The third point is what would make spatial analysis earn its place. A site-specific adjustment can only correct sites already in the training data. A spatial model could, in principle, estimate the offset for a new site from its neighbors and its catchment. If catchment covariates alone predict new-site offsets as well as the spatial model does, we will say so, and the recommendation reduces to covariate adjustment.
Several lines of prior work motivate this. Three metabolic signatures report strong internal results with no geographic context:
- the disulfidptosis signature in lung squamous carcinoma is a 9-gene score on 553 TCGA-LUSC samples with a 1-year C-index of 0.78. It assigns MEK inhibitors to high-risk patients and cisplatin to low-risk patients.
- the LAC-TME classifier combines lactate genes and immune cell scores in kidney cancer. It falls from a training C-index of 0.92 to 0.73 externally.
- the lactylation signature for breast cancer is a 7-gene score validated across 14 cohorts. It is reported to beat 86 other signatures.
None of them asks whether its validation cohorts differ geographically or operationally. On the health-geography side, the Iowa chemotherapy access study shows that rural patients travel roughly three times longer for chemotherapy. the CAR-T access study shows that facility alone changes the odds of receiving advanced therapy (OR 0.42 between two UC sites). Together they establish that site and place shape treatment and therefore survival.
Two pieces of prior work supply the methods.
- Spatial confounding test. Detecting Bias in the Presence of Spatial Autocorrelation and Evaluating Fairness in the Presence of Spatial Autocorrelation use eigenvector spatial filtering (ESF). ESF adds the main spatial patterns of a neighbor-weight matrix as regression covariates so that spatial confounding is removed. With it, they showed that an apparent demographic bias in Chicago bikeshare placement (p = 0.007) disappeared after spatial adjustment (p = 0.27). We reuse ESF to ask whether a signature's survival association survives the same kind of adjustment.
- Shift decomposition. Quantifying geographic domain shift splits cross-region transfer of mobility models into two parts. One is a feature-distribution shift, measured by mutual information, which hurt transfer. The other is a "Moran spatial shift," the difference in spatial clustering between source and target, which was positively associated with transfer. We borrow this decomposition for clinical survival models, where it has never been tested.
All models are local constrains the design. It argues that static external validation cannot protect deployments and that local revalidation is needed. Our goal is not to replace local validation but to predict which sites most need it.
Research Program
Geographic layer. For each TCGA tumor type studied, we decode the tissue source site from every sample barcode and map it to a contributing institution and its coordinates. We then attach catchment descriptors: state, rurality, and an area deprivation measure for the surrounding county. Some sites are biobanks or brokers whose patients' origins are unknown. These are flagged and analyzed separately rather than assigned a false location. For breast cancer we add METABRIC, whose patients come from a small number of distinct contributing centers, and the GEO cohorts used by the lactylation paper, located by submitting institution.
Re-implementation. Each published signature is rebuilt from its gene list and fitting procedure. We first confirm that our rebuild reproduces the reported in-sample numbers, because any later degradation is only interpretable relative to a faithful copy.
Four analyses.
- Site leakage. We fit a classifier that predicts the contributing site from the risk score and compare its accuracy with the same classifier using 1,000 size-matched random gene sets.
- Site- and region-held-out derivation. We repeat the full derivation pipeline, including the algorithm-combination screen, with whole sites or whole US census regions held out. All choices are locked before scoring the held-out data.
- Spatial structure of residual offsets. We estimate each site's residual survival offset with a shared-frailty Cox model. A frailty term is a per-site random multiplier on hazard. We then test the offsets for global Moran's I under inverse-distance and k-nearest-neighbor weights. Because the fairness paper warns that conclusions can hinge on the choice of weight matrix, results are reported across those weight choices.
- Shift regression. Across all derivation–validation pairs we can form, we regress the drop in C-index on feature shift and spatial shift, following the mobility paper.
Baselines
Naive reference: published random-split validation. This is the number each paper reports, and it is the reference point any reader currently trusts. The strongest reported figures are:
- a training C-index of 0.92 falling to 0.73 externally for LAC-TME;
- a 1-year C-index of 0.78 for the disulfidptosis signature;
- a claim of superiority over 86 signatures across nine cohorts for the lactylation signature.
No reported per-site C-index was found on Lacuna for any of these. We reproduce the published numbers first, then hold every analysis to the same gene lists, folds, and tuning budget.
Random gene sets of equal size. This is the "do nothing biological" baseline. If a random 9-gene score leaks site identity, or survives site-held-out validation, as well as the metabolic signature does, the metabolic label is adding nothing. Beating this baseline means the signature's site-held-out C-index sits above the 95th percentile of the random-set distribution.
Site-stratified Cox with catchment covariates. This is the strongest conventional competitor. Each site gets its own baseline hazard, and deprivation and rurality enter as covariates. It removes site-level offsets for every site seen in training, so for in-sample adjustment it is hard to beat. For predicting a new site's offset, its version is the covariate-only regression of site offset on catchment descriptors. The spatial model wins only if it lowers the leave-one-site-out error of the predicted offset beyond what covariates achieve. That comparison must use identical held-out sites and a fixed, prespecified set of weight matrices, with no tuning of the weights on held-out sites.
Ablation isolating the spatial ingredient. We keep the covariate model and add ESF eigenvectors. We then repeat the same comparison with the eigenvectors computed after randomly permuting site locations. If the permuted version helps just as much, the gain is model flexibility, not geography.
Strongest geography-agnostic multi-site method. From outside oncology, the OHDSI stroke model gives a sense of what multi-site validation without spatial terms achieves. It has a training AUC of 0.75, an external mean of 0.71, and collapses to 0.50 at an outpatient German site. It is a reference for the size of plausible drops, not a model we run.
Why not just…?
Why not just stratify by site and move on? For sites already in the training data, this is likely as good as anything we propose. We use it as the main in-sample adjustment, not as a straw man. Its limit is that it says nothing about a hospital that never contributed samples, which is exactly the deployment case All models are local worries about. The leave-one-site-out offset-prediction comparison is where this alternative either ties our method, in which case we report that spatial terms are unnecessary, or loses.
Why not just add distance or deprivation as Cox covariates? This is the real threat to the spatial component. With only tens of TCGA sites, Moran's I has little power, and a deprivation index may carry all the useful information. The bikeshare result in Detecting Bias in the Presence of Spatial Autocorrelation shows that ignoring spatial clustering can manufacture an association. But that was 2,080 block groups, not dozens of hospitals, so it does not establish that the same happens here. If covariates match the spatial model, the paper's conclusion becomes "geocode sites and adjust for catchment." That is still a change to current practice, but a smaller one.
Why not just do leave-one-site-out validation without spatial metrics? It is the core of analysis 2, so we do it. What it cannot do is explain or anticipate a failure. The OHDSI model's drop to AUC 0.50 was recorded but not diagnosed. The shift regression tests whether feature shift and spatial shift account for the drops, as they did for mobility models in the geographic domain shift paper. If neither predicts the drop, we report that plain held-out validation suffices.
Why not just use clinical stage instead of metabolic genes? Stage is included as a covariate in every model. The question here is not whether metabolic signatures beat stage. It is whether their validation is honest. That question matters most because these papers attach drug recommendations to the risk groups, for example MEK inhibitors versus cisplatin.
Proposed Experiments and Decisive Checks
The decisive test of the core claim has three possible outcomes.
- Claim supported. At least one signature loses a substantial share of its above-chance discrimination (C-index minus 0.5) under region-held-out derivation relative to random splits. It also predicts site of origin above the random-gene-set null.
- Claim refuted, signatures vindicated. All three signatures stay above the random-set 95th percentile under region-held-out validation and show no site leakage. This would be a positive result for metabolic stratification, and we would publish it as such.
- Spatial component tested. In leave-one-site-out prediction of site offsets, the ESF model should beat covariates alone, and the location-permuted ablation should lose that advantage. Residual offsets should show significant Moran's I under at least the prespecified primary weight matrix (inverse distance), with the other weight choices reported as sensitivity analyses.
A secondary check concerns the therapy recommendations. For each signature, we test whether the predicted drug-sensitivity contrast between risk groups changes when risk groups are assigned by the site-held-out model rather than the original. If group membership flips for many patients, the drug recommendations inherit the same instability.
Limitations stated up front. The sites are institutions, not patient homes, so catchment measures are ecological and may misattribute individual circumstances. Biobank-sourced samples cannot be placed. TCGA survival follow-up is itself uneven across sites, and that unevenness is part of what we are measuring rather than a defect we can remove.
References
- A novel disulfidptosis-related mRNA signature predicts prognosis and therapeutic response in lung squamous cell carcinoma
- LAC-TME classifier: machine learning-driven model predicts survival and prioritizes targeted therapy in clear cell renal cell carcinoma
- Personalized treatment decision-making using a machine learning-derived lactylation signature for breast cancer prognosis
- Access to Chemotherapy Services by Availability of Local and Visiting Oncologists
- Investigating CAR-T Treatment Access for Multiple Myeloma Patients Using Real-World Evidence
- Detecting Bias in the Presence of Spatial Autocorrelation
- Quantifying geographic domain shift to decouple the geospatial transferability of human mobility flow generation models
- All models are local: time to replace external validation with recurrent local validation
- Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network