Measuring Before Pooling: Invariance Testing for Pathology Biomarkers in Multi-Site Survival Models

When a tumor feature such as the fraction of stroma in a slide is used to predict survival, the reading comes from a pathologist's eye or from an image algorithm. In either case it is a measurement, and measurements can mean different things in different places. A pathologist trained in one country may score "stroma-high" at a different threshold than one trained elsewhere. A scanner or staining protocol can shift what an algorithm reports. Survival models that pool hospitals then treat the feature as if it had the same scale everywhere. When the estimated risk per unit of the feature differs across hospitals, nobody can say whether the biology differs or only the ruler does.

Psychologists who port questionnaires across cultures faced exactly this problem and solved it with measurement invariance testing. The procedure asks, step by step, whether several indicators of one hidden trait relate to that trait in the same way in every group before anyone compares groups. We propose to import that machinery into survival analysis of pathology biomarkers. The idea, stated in two sentences: treat each pathologist and each image algorithm as an "item" that measures a hidden biomarker, allow each item its own site-specific scale, and estimate the biomarker's effect on survival through that hidden variable rather than through any single reading. This separates two things current practice cannot tell apart. The first is observer and scanner disagreement, which is a property of the measurement. The second is genuine differences in prognosis between populations, which is a property of the patients.

Core Claim

The prognostic heterogeneity seen when pathology features are pooled across sites is partly a measurement artifact, and it can be removed by modeling readers and extractors as non-invariant indicators of a shared latent biomarker. We make three predictions. First, a joint model that combines a measurement layer with a Cox survival layer will estimate a per-site hazard ratio for the latent biomarker that is more homogeneous and better calibrated than observed-score Cox models. A Cox model, also called a proportional hazards model, relates covariates to the instantaneous event rate without specifying how that rate changes over time. Second, human visual scores that look prognostically useless on their own will regain a clear effect once their unreliability is modeled rather than ignored. Third, the indicators the invariance ladder flags as non-invariant (particular readers, particular scanners) will match independently known sources of disagreement.

Concretely, for patient $i$ at site $g$ with indicator $r$ (a pathologist's score or an algorithm's output), the measurement layer is

$$y_{igr} = \nu_{gr} + \lambda_{gr}\,\eta_i + \varepsilon_{igr},$$

where $\eta_i$ is the patient's latent biomarker level, $\nu_{gr}$ is an indicator- and site-specific offset (a reader who systematically scores high, a scanner that inflates stroma area), $\lambda_{gr}$ is the indicator's scale, and $\varepsilon_{igr}$ is reading noise whose variance encodes inter-observer unreliability. The survival layer is

$$h_i(t) = h_{0g}(t)\exp(\gamma_g\,\eta_i + \beta^\top z_i),$$

where $h_{0g}(t)$ is an unspecified site-specific baseline hazard, $z_i$ are clinical covariates such as stage and age, and $\gamma_g$ is the log hazard ratio of the latent biomarker at site $g$. The invariance ladder is then tested in order. Configural invariance asks whether the indicators load on one factor everywhere. Metric invariance asks whether the scales $\lambda_{gr}$ are equal across sites. Scalar invariance asks whether the offsets $\nu_{gr}$ are equal. Only after that ladder has been climbed does the scientific question become meaningful: is $\gamma_g$ equal across sites? If $\gamma_g$ differs under an invariant measurement layer, the difference is prognostic. If it differs only in observed-score models, it was the ruler.

Where this sits relative to prior work

Two papers motivate the gap by pairing automated pathology features with Cox models at a single site. Geessink et al. quantified tumor–stroma ratio with a segmentation network in 129 rectal cancer patients. Pathologist agreement was only κ = 0.578. The automated score kept a multivariable disease-specific-survival hazard ratio of 2.48, while visual assessment lost significance. They read this as evidence that human reads are noisy. Our model asks a sharper question: how much of that loss is attenuation from unreliability, which several readers combined through a latent variable could recover? Shapiro et al. trained a deep system with a Cox output layer on Graz colorectal cases. They showed that interpretable cluster features explain 73–80% of its risk score and that pathologists can learn a machine-found feature with 87.0–95.5% accuracy. They also explicitly call for geographically diverse validation. Their result supplies a candidate indicator (the tumor–adipose feature read by both humans and algorithms), but neither paper models the indicators jointly or across sites.

A third line supplies the cross-site setting without survival. Shamai et al. predicted receptor status from H&E across six cohorts in the US, Australia, Israel and Portugal and used disagreement with immunostaining to catch 31 misdiagnoses. That is exactly the kind of cross-scanner, cross-country measurement data our method needs, but it links nothing to outcomes. On the survival side, VLSA is the strongest whole-slide survival learner in the file, but it treats TCGA as one pool. Seidi et al. handle multi-site heterogeneity in federated Cox models, but the heterogeneity they address is which covariates each site records, not whether a shared covariate means the same thing at each site. Neither addresses measurement equivalence.

The method is borrowed from psychometrics. The four-level ladder used for the Chinese STAI, the Bergen Social Media Addiction Scale and by Luo et al. gives the testing order. Item-level bias detection of the kind Zhao et al. applied to PROMIS-29 gives the per-indicator diagnostic. One caution from that literature constrains us. Zhao et al.'s in-group derogation study shows that response-style differences can masquerade as trait differences. The pathology analogue is a reader with a systematically conservative threshold. Scalar non-invariance is meant to catch exactly this, but only if some indicator is trusted as an anchor, which we discuss below.

Research Program

The work has three parts.

The first part is a simulation study with known truth. We generate latent biomarkers, site-specific reader and scanner offsets and scales, reader noise matching reported agreement levels (κ around 0.58), proportional or deliberately non-proportional hazards, and censoring. We then compare how well each estimator recovers $\gamma_g$. This determines when the joint model helps and when it does not.

The second part is a multi-indicator dataset built on existing whole-slide survival cohorts. From the five TCGA survival benchmarks used by VLSA (over 2,800 patients), we take each tissue-source site as a group. For every slide we compute several automated indicators of a single interpretable biomarker, tumor–stroma ratio. These come from the same tissue segmented by independently trained networks and from stain-normalized and raw versions of each. For a stratified subset of a few hundred slides, we add visual tumor–stroma scores from four pathologists recruited from different countries' training systems. Reader training background thus becomes a "cultural group" in exactly the psychometric sense. The Graz split used by Shapiro et al. (1984–2007 versus 2008–2013) offers a temporal grouping if access can be arranged. Temporal drift in staining and practice is another non-invariance source.

The third part is estimation and diagnostics. We fit the joint model by maximum likelihood with a site-stratified partial likelihood, and in a Bayesian version for small sites. We climb the invariance ladder with likelihood-ratio tests and freed parameters. We report which indicators are non-invariant and whether $\gamma_g$ is homogeneous once the measurement layer is accounted for. Identification requires at least one anchor indicator assumed invariant. Our primary anchor is the stain-normalized segmentation area, since it is the most physically defined measure. We also run a sensitivity analysis that rotates the anchor across indicators, and we report results only when conclusions survive anchor choice.

Baselines

The naive reference is a pooled Cox model on a single observed automated score, with site-stratified baseline hazards and stage and age as covariates. This is what Geessink et al. did at one site, and it reported a hazard ratio of 2.48 for automated tumor–stroma ratio. Our first comparison is that model plus a feature-by-site interaction. This is the standard way an epidemiologist would test whether the effect varies, and it is the most direct competitor to our question. It is run on identical patients, covariates and folds.

The second baseline is regression calibration. For each patient we average all available readers and extractors, estimate reliability within site, correct the hazard ratio for attenuation, and allow site interactions. This is the strongest simple competitor for the "human reads are useless" claim, because it already undoes noise. It is given the same indicators as our model.

The third baseline is within-site standardization: z-scoring each indicator within site before pooling. This is the common harmonization habit.

For discrimination, the strongest learned competitor is VLSA, which reports an average concordance index (the probability that the model ranks a random pair of patients correctly by survival time) of 0.6954 across five TCGA benchmarks. We retrain it with the published setup and matched tuning budget, then evaluate it under leave-one-site-out splits. We do not expect our interpretable single-biomarker model to beat that concordance index, and beating it is not the claim. Our claim is judged on hazard-ratio recovery (bias and interval coverage in simulation), between-site heterogeneity of the biomarker effect, and leave-one-site-out calibration of predicted survival.

Two ablations isolate the key ingredient. The first keeps the joint model but forces full invariance, with equal $\lambda$ and $\nu$ everywhere. This isolates the value of modeling non-invariance as opposed to merely modeling noise. The second uses automated indicators only, with no human readers. This isolates the value of treating observers as items.

Our model beats the baselines if three things hold. In simulation it recovers $\gamma_g$ with nominal coverage where the interaction and calibration models do not. On real data it reduces the estimated between-site variance of the biomarker effect relative to the interaction model. And it improves leave-one-site-out calibration, all without losing more than a trivial amount of concordance relative to the observed-score Cox model.

Why not just…?

Why not just fit a stratified Cox model with a site interaction? Because the interaction coefficient mixes two things: a site where the scanner stretches the feature's scale, and a site where the biology carries different risk. Both produce a different per-unit hazard ratio. The interaction model detects heterogeneity but cannot attribute it. The first ablation (joint model with invariance forced) and the interaction baseline are fit to the same data. If our model's heterogeneity estimate is no smaller than the interaction model's, the measurement layer is not doing work on these data, and we will report that.

Why not just average several pathologists, or use expert consensus? This is a real threat for the within-site question. Averaging $k$ readers shrinks noise variance roughly by $1/k$ (independent errors average out), and regression calibration then undoes the remaining attenuation. If readers differ only in noise, this will nearly match our model, and the simulation will show exactly that. The joint model should win only when readers differ systematically in offset or scale, for example a conservative reader or a reader from a different training tradition. Averaging then passes that bias into sites whose reader mix differs. The reader study is designed so that reader composition varies across groups, which is where the two approaches should separate. If they do not, we will conclude that averaging suffices.

Why not just harmonize by stain normalization or by z-scoring within site? Stain normalization targets the scanner source of non-invariance but not reader thresholds. It enters our design as one indicator, so its sufficiency is directly testable: if the normalized score is already metric- and scalar-invariant, the ladder will say so. Within-site z-scoring does something worse. It forces every site to have the same mean biomarker level and so erases real differences in case mix. This is the psychometric error of imposing scalar invariance by fiat. We test it on TCGA sites with different stage distributions, checking whether z-scoring destroys between-site risk differences that the stage covariate confirms.

Why not just train a strong end-to-end survival model and ignore interpretable biomarkers? For ranking patients, a model like VLSA may well be better, and we report its concordance under site-held-out splits. But it answers a different question. Clinicians, trialists and guideline committees need to know whether a nameable biomarker read by humans carries the same risk in Lisbon as in Graz. An opaque score cannot answer that, and it inherits scanner shifts silently. Shamai et al. needed explicit multi-cohort engineering to survive scanner changes even for classification.

Proposed Experiments and Decisive Checks

The first decisive check is the simulation. We vary reader offsets, scales and noise across sites with the true $\gamma$ held equal, then separately make $\gamma$ truly unequal. The joint model should report homogeneity in the first case and heterogeneity in the second. The interaction model should report heterogeneity in both. If the joint model cannot make this distinction at realistic sample sizes (a few hundred patients per site, events in the tens), the method is not usable, and we will say so.

The second check is attenuation recovery. On the reader-study subset, we fit a Cox model on each single pathologist's visual score and on the latent variable estimated from pathologists only. Given agreement near κ = 0.578, we expect single-reader hazard ratios to be pulled toward 1. We expect the latent human-only estimate to approach the automated-score estimate. This directly tests whether the visual-assessment failure reported by Geessink et al. reflects unreliability rather than an absence of signal.

The third check is external agreement of the diagnostics. The indicators the ladder flags as non-invariant should coincide with independently recorded facts, such as scanner model, staining batch, and reader training background. As a measurement-only extension, we will ask whether H&E-predicted receptor status in the six Shamai et al. cohorts shows scalar non-invariance against the immunostaining anchor that tracks country or scanner. That cohort has no reported survival outcome, so this piece tests only the measurement layer.

The fourth check is leave-one-site-out transport. We fit on all sites but one and predict survival curves at the held-out site under each model. We report calibration slope and concordance, with the VLSA 0.6954 average concordance as the discrimination reference and the observed-score Cox model as the calibration reference.

Finally, because a latent effect can be homogeneous in hazard-ratio terms yet violate proportional hazards differently by site, we test proportionality of $\gamma_g$ over follow-up time within each site. Whether the joint model's conclusions depend on that assumption is reported as a limitation, not assumed away.