Background-Only Pseudo-Experiments for Osteosarcoma Prognostic Signatures

Particle physicists learned the hard way that a striking bump in data means little until you know how often the Standard Model alone, run through the same analysis, would produce one. They answer this by generating thousands of simulated "toy" experiments in which only known physics is present. They then ask how often the full search chain, including every choice of where to look, throws up an excess as large as the observed one. Osteosarcoma prognosis has the same structure and lacks this habit. The main public cohort, NCI TARGET-OS, has fewer than a hundred patients. It has been mined repeatedly for gene signatures that report very small p-values and high survival AUCs (the area under the curve for telling who will be alive at a given time). Yet no one has asked how often a generic, biologically meaningless gene set would do as well after passing through the same selection pipeline, or whether a signature adds anything beyond the known "background" drivers of outcome: metastasis at diagnosis, tumor purity, global methylation level, MYC-driven proliferation.

We propose to build that background and test against it. Concretely, we fit a "Standard Model of osteosarcoma outcome," a small survival model containing only established prognostic factors. We simulate pseudo-cohorts in which survival depends on those factors alone, while each patient's real expression and methylation profile is kept intact. We then rerun complete published signature pipelines on these pseudo-cohorts. The result is a calibrated global significance, and an incremental-value test, for any proposed osteosarcoma signature. It would also give the field a reusable way to judge new signatures before they consume scarce validation cohorts.

Core Claim

A signature's nominal p-value on TARGET-OS answers the wrong question. That question is "is this better than nothing?" The useful question is "is this better than what known prognostic factors plus an unrestricted search over genes would produce by chance?" We expect that, under background-only toys, standard pipelines will routinely produce log-rank p-values and time-dependent AUCs similar to published values. If so, those figures mostly measure the search, not the biology. Signatures that survive the test are worth prospective validation.

The new ingredient is the null hypothesis. Ordinary label permutation sets survival to be unrelated to everything. Our toys instead generate survival under a model containing the known prognostic factors. This mirrors how a collider search simulates Standard-Model-only data rather than empty data. The difference matters in osteosarcoma for a specific reason. Its genome is dominated by copy-number chaos, and proliferation and purity are correlated with thousands of transcripts. Many arbitrary gene sets should therefore "predict" survival simply by proxying these factors.

Grounding in Prior Work

Two physics papers supply the method. Killing the cMSSM softly is a global fit of a supersymmetric model to many experiments. It computes frequentist p-values by generating pseudo-experiments under the best-fit hypothesis rather than trusting asymptotic formulas. It also documents "dilution": adding weakly constraining observables changes the apparent p-value, which moved from (4.9±0.7)% to around 50% depending on whether one observable was included. We reuse three things from it: the toy-based p-value, the habit of reporting how a conclusion depends on which inputs are included, and the dilution diagnostic, applied to genes rather than observables.

Reinterpretation of LHC Results for New Physics is a community report. It argues that results are only reusable when full likelihoods and runnable analysis workflows are published. It motivates our deliverable of a public, rerunnable background model. We do not reuse any LHC-specific tooling.

The precision-spectroscopy work of Counts et al. plays a narrower, conceptual role. It shows that a deviation from the simple expectation must be split into the part known physics can produce (a smooth nuclear curve) and the part it cannot (a zigzag pattern) before anyone claims a new effect. We borrow only this rule of first explaining residuals with known effects; we do not port its mathematics.

On the oncology side, three papers motivate the proposal and serve as test cases. Qiu et al. built an eight-gene polyamine-metabolism signature on TARGET-OS and GEO data. Their pipeline used co-expression module screening, univariate Cox filtering and LASSO (a penalized regression that keeps only a few genes). They report AUCs of 0.872, 0.817 and 0.799 at 1, 3 and 5 years, and a risk-group split of p = 0.00012. This is exactly the kind of multi-stage search whose look-elsewhere effect has never been quantified.

Lietz et al. clustered TARGET-OS methylation profiles (n = 83). They report three risk groups (p = 5.0×10⁻⁴), with median recurrence-free survival ranging from 11.5 to over 100 months. They also report an odds ratio of 6.429 (p = 0.007) for chemotherapy non-response in hypermethylated tumors. They further claim that chemoresponse tracks global methylation while recurrence tracks site-specific CpG sites. That claim is a direct candidate for "does the local signal survive once the global background is in the model?"

Jiang et al. defined four multi-omic subtypes in 121 patients, including a MYC-driven subtype with under 40% five-year survival. This supplies one of our background covariates. The SGH-OS data may not be publicly available, so we treat that cohort as optional. Tu et al. showed that routine blood and clinical measures stratify risk. Their index is the natural clinical layer of the background model.

Research Program

The first component is the background model. On TARGET-OS we fit a Cox model whose covariates are limited to factors with prior support: metastasis at diagnosis, age, tumor site, estimated tumor purity, global mean methylation, a MYC-amplification or MYC-expression indicator, and a generic proliferation score. We include Tu-style clinical variables where TARGET-OS records them. Covariates are fixed in advance and never selected on outcome. We fit the censoring distribution separately.

The second component is the toy generator. For each of $B$ pseudo-cohorts (on the order of 10⁴), we keep every patient's real molecular profile and covariates. We then draw a new survival time from the fitted background model and a new censoring time from the censoring model. In every toy, any gene's association with outcome arises only through its correlation with the background factors, exactly as in reality if no new biology exists.

The third component is pipeline replay. We reimplement each published pipeline end to end, including every data-dependent choice: module screening, univariate filter threshold, LASSO penalty chosen by cross-validation, and median or optimal cutpoint. Each pipeline is run on every toy. For a pipeline with test statistic $T$ (for example, the log-rank chi-square of the final split, or the 3-year AUC), the global p-value is

$$p_{\text{global}} = \frac{1 + \#\{b : T_b \ge T_{\text{obs}}\}}{1 + B},$$

where $T_b$ is the statistic the full pipeline produces on toy $b$ and $T_{\text{obs}}$ is the published value reproduced on the real data. Because the pipeline searches anew on every toy, this number automatically includes the trials factor of the gene search.

The fourth component is incremental value and dilution. On the real data we measure the gain in concordance (C-index, the probability that the model correctly orders two patients' survival times) from adding a signature to the background model. We compare that gain with the distribution of gains from size-matched random gene sets. We also compare it with gains from sets drawn to match the signature's correlation with proliferation. Following the dilution analysis in the FITTINO work, we also report how $p_{\text{global}}$ moves as individual genes or background covariates are dropped. This identifies which elements carry the result.

The deliverable is a public package. It contains the fitted background model, the toy generator, and replayable pipelines for TARGET-OS and the GEO osteosarcoma sets. With it, any new signature can be tested against the same background before external validation.

Baselines

The naive reference point is the nominal statistic as published. This covers Qiu et al.'s log-rank p = 0.00012 and AUC of 0.817 at three years, and Lietz et al.'s cluster p = 5.0×10⁻⁴ and odds ratio of 6.429. We first reproduce these on the same TARGET-OS data. Any discrepancy is reported before toys are run, so differences are not blamed on preprocessing.

The strongest existing competitor for "does this signature carry information?" is full-pipeline label permutation. Survival labels are shuffled and the entire selection pipeline is rerun. This is the standard careful fix in biostatistics, and it already captures the trials factor. It is run with the same number of replicates, the same pipeline code and the same compute as our toys, so the only difference is the null hypothesis. Our method beats it if, for at least one published signature, permutation declares significance while background-only toys do not. A further requirement is that the background-only verdict agrees better with external validation performance on held-out GEO and methylation cohorts. If the two nulls always agree, our key ingredient adds nothing and we will say so.

A second baseline is the background model alone, without any signature. It is evaluated by C-index and time-dependent AUC on TARGET-OS with nested cross-validation, and on the external cohorts where covariates exist. For the clinical-index component, we found no reported numeric result on Lacuna for the Tu et al. index. Its performance will therefore be measured, not quoted. A signature counts as adding value only if its increment over this model exceeds the 95th percentile of increments from random and proliferation-matched gene sets.

A third baseline is nested cross-validation of each published pipeline, with all selection inside the training folds. This is the usual guard against optimistic AUCs. We report it next to toy results because it corrects optimism but does not adjust for background confounding.

The ablation that isolates our key ingredient varies the background model. We run the toys with progressively smaller background models: none (equivalent to permutation), clinical factors only, and clinical plus molecular background. We then track how each signature's global p-value changes. If the molecular background terms do not move the verdicts, our motivating hypothesis about proliferation and purity proxies is wrong.

Why not just…?

Why not just run a permutation test through the whole pipeline? This is the real threat. For signatures that are pure noise it will reach the same conclusion. It fails in one specific case: a signature that is a disguised proliferation or purity score. Permutation correctly finds that such a signature predicts survival and wrongly implies that the specific genes matter. That is precisely the case that misleads target-discovery work in a cancer without recurrent driver mutations, as Jiang et al. emphasize. The ablation above settles whether this case occurs in practice. If permutation and background-only toys never disagree on the test signatures, the simpler method suffices.

Why not just validate externally? External validation is the final arbiter, and we use it. But the osteosarcoma validation cohorts are, in Lietz et al.'s words, small independent datasets, and they lack reported per-cohort survival numbers. With such samples, a failed validation cannot separate a false signature from an underpowered test. The toys also give the power needed for that interpretation. They tell us, for each signature, what external effect size a true signal would produce and how many patients would be needed to see it.

Why not just use penalized Cox with nested cross-validation? Nested cross-validation removes optimism from the selection step. It still benchmarks against zero rather than against known prognostic factors. It also gives no global p-value covering the many signatures already extracted from the same cohort. We report it as a baseline, so readers can see where it and the toys diverge.

Why not just use clinical indices or post-surgery necrosis? Necrosis arrives only after months of chemotherapy, as Lietz et al. note. Clinical indices ignore treatable biology such as homologous-recombination deficiency, the impaired DNA-repair state that responded to combined cisplatin and olaparib in Jiang et al. We do not compete with these factors; we fold the clinical ones into the background, so any molecular signature must earn its place on top of them.

Decisive Checks

The first check is calibration. Within the toys, a pipeline-level test at the 5% level should reject about 5% of the time when we fit and simulate from the same background model. If the nominal log-rank tests used in the literature reject far more often, say in most toys, then published nominal p-values on TARGET-OS are uninformative. That is a publishable finding on its own.

The second check is a re-evaluation of the published signatures: Qiu et al.'s eight genes, and Lietz et al.'s global and site-specific methylation groups. For each we report $p_{\text{global}}$, its percentile among size-matched random sets, and its incremental C-index over the background model. A clear prediction follows for Lietz et al.'s split. If chemoresponse is truly a global-methylation effect while recurrence is a local one, the site-specific CpG signal should keep its significance with global methylation inside the background, while the chemoresponse signal should be absorbed by it.

The third check closes the loop. We test whether ranking signatures by $p_{\text{global}}$ predicts their held-out performance on GEO and the independent methylation cohorts better than ranking by nominal p-value or by permutation p-value. That result decides whether the physics-style background null should replace current practice or merely sit alongside it.