Surgeons and anesthetists want to know which combinations of risk factors make an older patient likely to become delirious after surgery. Examples would be poor baseline cognition combined with swinging blood pressure, or high glucose combined with low lymphocytes. Almost every such claim in the postoperative delirium literature comes from one hospital and has never been tested anywhere else. This proposal borrows a simple tool from geography, the "geographical detector." It measures how much of the variation in an outcome is explained by sorting cases into groups defined by one or two factors. We make one change to it: interactions are discovered in one set of hospitals and scored in hospitals that played no part in the discovery.
The key observation is almost embarrassingly simple. For a yes/no outcome such as delirium, the detector's statistic $q$ is exactly the Brier skill score of a lookup table that predicts each patient's risk as the delirium rate of their group. Its best-known failure is that crossing many factors produces tiny cells and pushes $q$ toward 1. That failure is plain in-sample overfitting. Scoring the locked table on external patients removes it and turns interaction detection into an external validation problem. The result is a concrete, auditable test: an interaction is real only if the frozen two-factor table beats both single-factor tables in new hospitals.
Core Claim
Wang's $q$-statistic, as implemented in the geographical-detector toolbox gdverse, is defined as $q = 1 - \mathrm{SSW}/\mathrm{SST}$. Here SSW is the within-group sum of squared deviations of the outcome and SST is the total sum of squares. When the outcome $y_i$ is 0/1 and $\hat p_{h}$ is the observed delirium rate in group $h$, SSW is the total squared error of predicting $\hat p_{h(i)}$ for every patient $i$, and SST is the squared error of predicting the overall rate. We therefore define the external version as
$$ q_{\text{ext}}(S) = 1 - \frac{\sum_{i \in E} \big(y_i - \tilde p_{h(i)}\big)^2}{\sum_{i \in E} \big(y_i - \bar y_E\big)^2}, $$
where $S$ is a stratification (one factor, or the crossing of two factors) whose cutpoints and group rates were fixed on derivation data, and $E$ is the set of external patients. The term $\tilde p_{h}$ is the derivation rate of group $h$ after a single intercept shift on the log-odds scale is fit in $E$, which absorbs site-level differences in base rate. The term $\bar y_E$ is the external base rate. Unlike the in-sample $q$, $q_{\text{ext}}$ can fall below zero, which honestly signals a stratification that is worse than knowing nothing.
The detector's interaction categories are then applied to external scores. A pair of factors is called "bi-enhanced" only if $q_{\text{ext}}(X_1 \cap X_2) > \max(q_{\text{ext}}(X_1), q_{\text{ext}}(X_2))$ with a bootstrap interval that excludes zero. It is called "nonlinear-enhanced" only if the external joint score exceeds the sum of the two single-factor scores.
Our claim is twofold. First, a substantial share of delirium interactions that look strong in-sample, whether by in-sample $q$, by SHAP interaction values, or by logistic product terms, will not survive this test. Second, a compact model built only from interactions that replicate will lose little external discrimination relative to a large gradient-boosted model, and will be better calibrated. This matches the finding of Spathis and Hyland, who showed across 127 eICU hospitals that smaller models transport better than overparameterized ones.
Why This Gap Is Real
The file supplies three kinds of motivating evidence. The first concerns single-center work. Every modern delirium model in the file was built and tested inside one institution. This includes the UCSF gradient-boosting model of Bishara et al., the Charité intraoperative transformer TRAPOD of Giesa et al., the Zurich SHAP-clustering phenotypes of Zheng et al., and the Dalian and Qingdao nomograms of Liu et al. and Lin et al.. The two nomograms call a 7:3 random split "validation." Several of these authors explicitly ask for external testing.
The second concerns interactions. Bishara et al. attribute the gap between their gradient-boosted trees and linear scores to higher-order interactions, but they never name or test them. Zheng et al. cluster additive SHAP attributions and so never test a joint stratification either.
The third concerns what goes wrong outside the derivation hospital. Wang et al. found that a locked lasso model held up in most OHDSI databases but fell to chance in a German outpatient database. Spathis and Hyland warn that apparent hospital-to-hospital shifts are often small-sample noise. That warning shapes our design: we report intervals and pool across hospitals rather than trusting any single small site.
The geographical detector itself supplies the method. The AED equity study shows its optimized-discretization variant, OPGD, producing interaction rankings in two cities. Notably, the two cities disagreed on which pair mattered most: residential by company density in Guangzhou, but GDP by residential density in Shenzhen. That disagreement is exactly the kind of non-replication our procedure is designed to quantify rather than narrate. The toolbox also documents the constraints we inherit. Crossing strata makes cells sparse and inflates $q$, and $q$ carries no sign. We address the first by scoring out-of-sample and the second by always reporting the table of group rates alongside $q$.
Research Program
We will build a small, open pipeline. It will take a candidate list of at most about ten perioperative factors, discretize each with OPGD-style optimization on derivation data only, and lock both the cutpoints and the group rates. It will then score single-factor and pairwise-crossed tables with $q_{\text{ext}}$ in held-out hospitals.
The candidate factors come from the clinical literature in the file. They are age, prior dementia or cognitive impairment, preoperative glucose and lymphocyte count (Liu et al.), and systolic blood pressure variability in the first postoperative day (Shen et al.). The list also includes opioid and sedative exposure, surgery type, and a severity score. We will not search over hundreds of features. The point is to test clinically named hypotheses, and the sparsity problem grows with the number of factors crossed.
Group-rate estimation will include one optional refinement, which we treat as an ablation. Cells with few derivation patients will be shrunk toward the rate implied by their parent single-factor groups, so that a three-patient cell cannot dominate the score.
The primary data are postoperative ICU admissions in the multi-hospital eICU database, evaluated leave-one-hospital-out and pooled over hospitals as in Spathis and Hyland. MIMIC-IV, from a separate health system, serves as a fully external cohort and will be processed with the MIMIC-IV pipeline.
Labels are the main feasibility risk. Chen et al. show that diagnosis codes capture under 3% of delirium cases, so codes are unusable as labels. We will use recorded bedside delirium screens (CAM-ICU-type assessments) wherever they exist. As a secondary label in MIMIC-IV, we will use note-derived delirium symptoms following Chen et al.'s clinical-language-model approach. Step one of the project is a label audit: per hospital, the fraction of surgical admissions with any recorded screen and the positive rate. Hospitals with no screening are excluded rather than imputed. If eICU screening coverage proves too thin, the study falls back to MIMIC-IV plus any single-center cohort a collaborating group can share under the same locked protocol.
Baselines
The naive reference point is the external base rate with an intercept only. It defines $q_{\text{ext}} = 0$ and an AUROC of 0.5. Any interaction or model must beat it, and in small hospitals the bootstrap interval around zero shows how much apparent skill is noise.
The first competitor for interaction discovery is logistic regression with a flexible spline for each factor plus a single product term per pair. Pairs are ranked by derivation likelihood-ratio statistics, and the locked model is scored externally on the same patients with the same intercept-only recalibration. It is the strongest classical competitor because it is exactly what a clinical statistician would do. The nomograms of Lin et al. and Liu et al. are its additive special case.
The second discovery competitor is XGBoost with TreeSHAP pairwise interaction values. Pairs are ranked by mean absolute interaction value in derivation, following the modeling style of Bishara et al. and Zheng et al.. The third is the in-sample $q$ typology exactly as gdverse computes it, which serves as the ablation isolating our one change, out-of-sample scoring. All three competitors see the same derivation hospitals, the same candidate factors, and the same hyperparameter-search budget measured in configurations tried. Each produces a ranked list of pairs. Discovery methods are compared on the fraction of their top-ranked pairs that meet the external bi-enhanced criterion, and on how many planted interactions they recover in simulation.
For prediction, the strongest existing method is a full-feature XGBoost model. Bishara et al. report an AUC-ROC of 0.851 for XGBoost on 115 preoperative features at UCSF, measured on an internal split, against 0.762 for the AWOL-S score. For intraoperative time series, Giesa et al. report an AUROC of 0.774 and an AUPRC of 0.330 for TRAPOD. I found no reported result on Lacuna for any postoperative delirium model evaluated on an external cohort. The nearest external-transport figures are OHDSI's internal AUC of 0.75 against an external mean of 0.71 for stroke hemorrhagic transformation. We will retrain XGBoost on our derivation hospitals with the same tuning budget rather than import the UCSF number. Our compact model, a logistic model using only the replicated stratified interactions, counts as successful if three conditions hold on held-out hospitals. Its pooled external AUROC must fall within 0.02 of XGBoost's. Its calibration slope must be closer to 1. Its hospital-to-hospital spread in AUROC must be no wider.
A further ablation removes shrinkage of sparse cells, isolating whether that refinement matters or whether external scoring alone suffices.
Why not just…?
Why not just add product terms to a logistic model and test them? This is the most serious threat. With a single pair and a spline basis, a likelihood-ratio test and a crossed lookup table ask closely related questions. The file has no evidence that product terms replicate worse. We therefore make it a head-to-head comparison. If spline-plus-product-term logistic regression replicates as often as external $q$, we will report that the contribution is the out-of-sample scoring protocol and the readable group-rate table, not the $q$-statistic itself. The protocol applies unchanged to either statistic.
Why not just read interactions off SHAP values from a boosted model? SHAP interaction values describe how one fitted model combines features. They can reflect site-specific documentation patterns just as easily as physiology, and Zheng et al.'s phenotypes were never checked outside Zurich. SHAP is included as a discovery baseline. The simulation with planted and absent interactions under a simulated site shift will show whether it produces more false replications.
Why not just externally validate XGBoost and skip interactions altogether? This answers a different question, whether the whole model transports, and says nothing about which clinical combinations to act on. It is also the prediction baseline we must match. If XGBoost's external AUROC exceeds the compact model's by well over 0.02, we will say so plainly and conclude that transportable delirium risk is not concentrated in a few pairwise interactions.
Proposed Experiments
The first check is a simulation that uses real eICU covariates with synthetic outcomes. We plant known interactions of each detector category, and we plant spurious ones that exist only in some simulated "hospitals." We then measure false and true replication rates for in-sample $q$, $q_{\text{ext}}$, product-term regression, and SHAP as the number of strata per factor grows. The prediction is that in-sample $q$ labels more and more pairs as enhanced as cells get sparse, while $q_{\text{ext}}$ does not.
The second check is the real replication study. We derive on a random half of eligible eICU hospitals and score leave-one-hospital-out on the rest, then on MIMIC-IV. We report, for each discovery method, how many of its top ten pairs replicate as bi-enhanced with intervals excluding zero. Following Spathis and Hyland, we also test whether hospital-level disagreement exceeds what resampling within a pooled cohort would produce.
The third check compares the compact replicated-interaction model with retrained XGBoost and an additive nomogram. Metrics are pooled and per-hospital AUROC, AUPRC, Brier score, and calibration slope.
The fourth is a label sensitivity analysis in MIMIC-IV, repeating the replication study with screen-based and note-based labels. It shows how much apparent non-replication comes from outcome definition rather than biology, which is the measurement problem flagged by the spread in reported incidence, from 5.3% at UCSF to 23.8% in Dalian.
The project succeeds if external scoring clearly separates replicating from non-replicating interactions and the compact model transports as well as the black box. It fails informatively if either no interaction replicates or ordinary product terms do just as well.
References
- gdverse: Analysis of Spatial Stratified Heterogeneity
- Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
- Postoperative delirium prediction using machine learning models and preoperative electronic health record data
- Applying a transformer architecture to intraoperative temporal dynamics improves the prediction of postoperative delirium
- Clustering of Disease Trajectories with Explainable Machine Learning: A Case Study on Postoperative Delirium Phenotypes
- A clinical prediction model for postoperative delirium following surgical treatment of femoral fractures in older adult patients: development and validation study
- Development and validation of a postoperative delirium risk prediction model for non-cardiac surgery in elderly patients
- Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network
- Observed Equity and Driving Factors of Automated External Defibrillators: A Case Study Using WeChat Applet Data
- Perioperative blood pressure variability as a risk factor for postoperative delirium in the patients receiving cardiac surgery
- An Extensive Data Processing Pipeline for MIMIC-IV
- Identifying Symptoms of Delirium from Clinical Narratives Using Natural Language Processing