Hospitals routinely choose between candidate prediction models (a logistic regression, a gradient-boosted tree, a recurrent network over vital signs) using validation data from their own patients, and then deploy the winner at sites whose patients and record-keeping differ. The model that wins at home often loses abroad. The Epic Sepsis Model fell from a vendor-reported AUC of 0.73–0.83 to 0.60–0.64 at external hospitals (All models are local). In the OHDSI stroke study, one model averaged 0.71 AUC across ten external databases but scored 0.50, chance level, on a German outpatient database that lacked acute in-hospital procedures (OHDSI hemorrhagic transformation model). Patient-level external data is usually locked behind privacy rules. Coarse descriptions of an external site are often available, though: its published cohort table, its data dictionary, its care setting, how often it orders labs, and its outcome prevalence.
This note proposes using those scattered, non-patient-level descriptions to build a synthetic stand-in for an unseen external site. The question is whether that stand-in correctly predicts which candidate model will rank highest there. It does not need to predict exact accuracy. The synthetic cohort is made by applying explicit structural changes to the developer's own data, such as dropped variables, changed measurement frequency, shifted prevalence, and truncated care settings. A small correction step is then learned from other sites where the true external result is known. The two-sentence version is this: assemble a profile of the target hospital from heterogeneous public summaries, transform internal patients to match it, and rank candidate models on the result. We test whether this ranking tracks real external rankings better than internal validation, simple reweighting, and a small labeled sample from the target site.
Core Claim
Model selection for transport needs only rank preservation, not accuracy parity. This was shown for in-distribution image validation by Synthetic data for model selection, which reweights synthetic samples by ridge regression over about five anchor models and reaches a Spearman correlation of 0.986 with real ImageNet accuracy. Our claim is that the same idea extends to cross-hospital transport if two things change.
First, the synthetic data must contain the kinds of shift that actually break clinical models. The worst failures are changes in support, where variables are missing or a care setting is not represented at all, not merely changes in proportion. Reweighting internal patients cannot produce a patient type the internal data never contains. Second, the correction step cannot use anchors measured on the target site, because the target's labels are what we lack. Instead it is learned across other sites, leaving the target site out.
The specific new result would be this: a rank forecast built only from aggregated site descriptions beats internal cross-validation at choosing the externally best model. Its advantage should come from the support-changing operators, and should be concentrated on sites with setting mismatch like the German OHDSI case.
Relation to Prior Work
Two papers already connect synthetic data to external validation, and they define what this proposal must do differently.
Bridging the Generalisation Gap builds parametric synthetic "sites." Features come from Normal, Uniform and Categorical draws, outcomes from a logit risk model, and prevalence is tuned per site. It finds that tree ensembles lose AUROC under shifted interactions while logistic regression stays stable. It never ingests real heterogeneous records, and it never checks its simulated failures against real external cohorts. We start from real multi-table ICU records, and the whole evaluation is against real held-out sites.
Synthetic Data as Validation inserts procedurally generated tumors into healthy CT volumes pooled from three sources and reports better out-of-domain results on FLARE'23. Dice rose from 31.1% to 35.4%. This method is specific to organ imaging, and it uses synthetic data to train and validate a single model rather than to forecast how different models compare at a named site.
Several other papers motivate the design. The fidelity–validity dissociation in Beyond Statistical Fidelity is one: generators with low distributional distance still produced zero ontologically valid records. This warns us against neural tabular generators and pushes us toward constrained transformations of real records. Spot the difference found that synthetic derivatives reproduce within-network analyses but censor rare outliers. That is exactly the tail our support-changing operators must preserve rather than smooth away.
The apparent contradiction between "synthetic validation generalizes" and "all models are local" (All models are local) becomes testable here. If site-local operational differences can be written down as structural operators, a forecast can capture them. If they cannot, the forecast fails, and the local-validation camp is right for this setting.
The mixed evidence on model complexity also becomes testable. Logistic regression wins under synthetic shift in the parametric study. A two-level attention LSTM beats logistic regression externally, 0.855 versus 0.741 AUC from eICU to MIMIC-III (ICU mortality with external validation). Gradient boosting beats logistic regression across two London hospitals, 83% versus 71% accuracy for severe bleeding (ALGIB gradient boosting). A correct rank forecast should reproduce each of these outcomes from site descriptions alone.
Research Program
Site profiles. For each target site we assemble a profile from heterogeneous, non-patient-level sources. These are the marginal summaries of the kind found in a published cohort table, the list of variables present in its data dictionary, per-variable measurement frequency and missingness, unit-type and care-setting descriptors, coding vocabulary, and outcome prevalence. The aggregation problems named in the multi-source literature apply directly here: entities with different names across sources and conflicting summaries. We handle them by mapping every profile onto the MIMIC-Extract set of 104 clinical aggregates (MIMIC-Extract), with ICD and drug harmonization following the MIMIC-IV pipeline.
Structural shift operators. We define a small library of transformations of internal patient records, each with a parameter set from the profile:
- removing a variable the target does not record;
- thinning or densifying a measurement time series to match the target's ordering frequency;
- moving the decision threshold to hit the target's prevalence, borrowing the prevalence-matching mechanism of the parametric framework;
- restricting or reweighting by acuity and unit type;
- re-coding diagnoses into the target's vocabulary.
Composing these operators yields a synthetic proxy cohort, $\tilde{D}_s$, for site $s$. Because every record is a transformed real record, ontological validity is inherited rather than learned.
Cross-site calibration. Let $m$ index candidate models. Let $\tilde{a}_{m,s}$ be model $m$'s AUC on the proxy for site $s$, and $a_{m,s}$ its true external AUC. On a set of training sites whose labels we do hold, we fit a correction $g$ to predict $a_{m,s}$ from $\tilde{a}_{m,s}$, the internal AUC, and simple model descriptors such as model family and number of inputs used. This step adapts the ridge calibration of Synthetic data for model selection, with sites in place of anchor models. We then apply $g$ to a new site that contributed no labels.
The quantity we care about is rank agreement, $\rho_s = \text{Spearman}(\{g(\tilde{a}_{m,s})\}_m, \{a_{m,s}\}_m)$, together with selection regret, $R_s = \max_m a_{m,s} - a_{\hat{m},s}$. Here $\hat{m}$ is the model the forecast picks.
Candidate pool. About 60 models, spanning Lasso and logistic regression, random forest, gradient boosting, GRU-D, LSTM, and the two-level attention LSTM, each at several hyperparameter settings and input subsets. The tasks are in-hospital mortality and vasopressor/ventilation onset, as defined by MIMIC-Extract.
Baselines
The naive reference is internal validation. We rank models by their cross-validated AUC on the development data and pick the top one. This is what most groups do. The ICU mortality study quantifies its blind spot: internal AUC was 0.912 against an external 0.855. The proxy beats this baseline if its mean Spearman correlation across held-out sites is higher and its mean selection regret is lower. Significance is assessed with a paired bootstrap over sites.
The strongest simple competitor is marginal reweighting. Internal patients are importance-weighted so that their feature marginals and prevalence match the target profile, and models are ranked on the reweighted internal set. This uses exactly the same profile information as our method, with none of the structural operators. It receives the same cross-site correction $g$ and the same tuning budget.
The strongest real-data competitor is local validation with a small labeled sample, as advocated by All models are local. Models are ranked on $k$ randomly drawn labeled target patients, for $k$ from 50 to 1,000. We report the $k$ at which this baseline matches the proxy, which is an exchange rate between synthetic profiles and real labels. That parallels the 30:1 frame-equivalency ratio measured for tracking in On the Equivalency, Substitutability, and Flexibility of Synthetic Data.
A fourth baseline is the parametric simulator of Bridging the Generalisation Gap, fitted to each site's profile.
The ablation that isolates the key ingredient removes each operator family in turn, and removes the cross-site correction.
For the strongest prior method on the same benchmark, the file reports no clinical rank-correlation numbers. The closest reported figure is Spearman 0.986 for calibrated DiT validation on ImageNet (Synthetic data for model selection), which is an in-distribution vision result. No reported result was found on Lacuna for synthetic forecasting of external clinical model rankings.
Why not just…?
Why not just reweight internal data to match the target's marginals? This is the real threat, and we treat it as one. When the target's patients are a reshuffled version of the developer's, reweighting should do as well as our operators, and the ablation will show this. Reweighting cannot represent a variable that is absent at the target, or a care setting with no counterpart at home. The OHDSI German collapse was a mismatch of exactly that kind. Our prediction is therefore specific: the proxy's advantage over reweighting should appear only on held-out sites with support mismatch. If reweighting matches the proxy on those sites too, we will report that the structural operators are unnecessary.
Why not just collect a few hundred labeled patients at the target? Where governance allows it, this is the right answer, and the local-validation argument says so. Our experiment measures how many labeled patients the proxy is worth. If that number is under about 100, the proxy is not worth its engineering cost and we will say so. Its value is concentrated in pre-deployment decisions, where even a small labeled sample is not yet obtainable.
Why not just always deploy logistic regression, since it transports most stably? The real evidence disagrees with this as a blanket rule. The attention LSTM beat logistic regression externally, 0.855 versus 0.741, and gradient boosting beat it across two London hospitals. A fixed rule wins on some sites and loses on others. We include "always the best linear model" as a baseline and report its regret per site.
Why not just train a tabular GAN or diffusion model on internal data? Such generators produced zero valid records under ontology checks in Beyond Statistical Fidelity. Even when valid, they reproduce the internal distribution and say nothing about a different site. We include an internal-trained neural generator as a negative control.
Proposed Experiments
The main testbed is the eICU Collaborative Research Database, with more than 198,000 encounters across many US hospitals. We hold out each hospital that has enough mortality events, building its profile from aggregated summaries only, and calibrate on the remaining hospitals. Profiles are computed at the hospital level so that no patient-level information leaks into the forecast.
The second testbed is the published eICU→MIMIC-III transport pair. Here the forecast must reproduce the known ordering of the attention LSTM above logistic regression.
The third is a constructed setting-mismatch stress test within MIMIC-IV, in the spirit of the OHDSI German failure. Models are developed on ICU stays and evaluated on non-ICU hospital admissions from the hosp module, where many ICU variables are absent.
The metrics throughout are per-site Spearman correlation, top-1 selection regret in AUC points, and absolute error in predicted external AUC. We also stratify by subgroup to check whether the forecast catches the fairness gaps flagged in the MIMIC-IV pipeline.
Four results would decide the question:
- Across held-out eICU hospitals, the proxy should beat internal validation on both rank agreement and regret.
- On the MIMIC-IV setting-mismatch test, the proxy should beat marginal reweighting by a clear margin. Removing the variable-removal and measurement-thinning operators should erase that margin.
- The labeled-sample exchange rate should exceed a few hundred patients.
- The forecast should not invert the known eICU→MIMIC-III ordering.
If result 1 holds and result 2 fails, the contribution shrinks to "profile-based reweighting suffices," which is still a useful negative finding. If result 1 fails, the evidence supports recurrent local validation over any synthetic substitute for this task.
References
- All models are local: time to replace external validation with recurrent local validation
- Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network
- Synthetic data for model selection
- Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation
- Synthetic Data as Validation
- Beyond Statistical Fidelity: Causal-Valid Synthetic EHR Generation for Low-Resource Clinical AI
- Spot the difference: comparing results of analyses from real patient data and synthetic derivatives
- A Clinically Practical and Interpretable Deep Model for ICU Mortality Prediction with External Validation.
- Prediction of Outcome in Acute Lower Gastrointestinal Bleeding Using Gradient Boosting
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- An Extensive Data Processing Pipeline for MIMIC-IV
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data