Choosing Clinical Models by Social Context, Not by Leaderboard

Hospital risk models are usually picked the way benchmark models are: train several candidates, keep the one with the best overall discrimination (how well it ranks sick patients above healthy ones), and deploy it to everyone. Social determinants of health are the non-clinical facts that shape outcomes, such as housing, income, transportation, and neighborhood deprivation. They lose under this rule almost by construction. When community-level social features are linked to ICU records, population discrimination does not move, but calibration (whether a predicted 20% risk really means 20%) improves sharply for in-hospital mortality, and false-positive gaps for vulnerable intersectional groups shrink (Evaluating the Impact of Social Determinants on Health Prediction). A selector that looks only at pooled discrimination discards these features as uninformative.

We propose to treat the decision "which model, with which social information, for which patients" as a context-dependent selection problem that is made separately for each care setting and each region of social-vulnerability space. The decision has stated uncertainty, and it is scored on worst-group calibration subject to discrimination being practically unchanged. The menu deliberately includes the cheapest competitor, a clinical-only model recalibrated within social strata. The first scientific question is therefore whether social features help these models by changing who ranks above whom or only by shifting how risk levels are scaled within groups. That answer determines what a hospital should actually build.

Core Claim

The "accuracy-neutral, fairness-active" pattern has two possible explanations, and they lead to different engineering choices. Under the first, social features act as a grouping variable. They tell the model which stratum a patient belongs to, so the model can correct a stratum-level calibration offset, and a clinical model with per-stratum recalibration would do as well. Under the second, social features change the ordering of patients within strata. That requires an enriched model, and the benefit should appear as within-stratum discrimination gains concentrated in high-deprivation regions.

We claim that which explanation holds depends on context, meaning the task and setting and the patient's social stratum, and that a selector which estimates this per context will beat both "always include social features" and "never include them" on worst-stratum calibration at equal population discrimination. The direction of subgroup error is already known to flip across settings. Public-insurance patients see higher error in ICU mortality (zero-one loss about 0.198 vs 0.158), while private-insurance patients see higher error in psychiatric readmission (about 0.22 vs 0.20) (Can AI Help Reduce Disparities). This is the motivating evidence that a single global decision is the wrong unit.

Research Program

The candidate menu has five arms, each trained with the same tabular learners the prior work used (logistic regression, random forest, gradient-boosted trees). Arm 1 is clinical-only. Arm 2 is clinical-only with isotonic recalibration fitted separately in each social stratum. Arm 3 adds community-level features from County Health Rankings, the Social Vulnerability Index, and the AHRQ database, following the linkage of Yang et al.. Arm 4 adds individual-level social features extracted from discharge notes by a classifier trained on MIMIC-SBDH. Arm 5 is a stratum-decoupled model with stratum-specific thresholds, in the style of Hu et al..

Arm 4 is included because individual-level needs might change rankings where zip-code averages cannot. Contextual indices alone reached only AUROC 0.54 for lupus flares (Li et al.). The note extractors, however, are noisy: they fail 44.5% of negation tests and 70.9% of education-attribution tests. We will therefore also run Arm 4 with extractor outputs perturbed at those error rates, to see whether any ranking benefit survives realistic extraction noise.

Contexts are defined as task (in-hospital mortality, 30-day readmission, 1-year mortality) crossed with insurance type and Social Vulnerability Index quartile. We keep strata coarse on purpose, so that each has enough events to measure calibration. Selection uses the hierarchical Bayesian comparison of Szymanski & Gorman, originally built to show that claimed differences between part-of-speech taggers often collapse into practical equivalence. We reuse its method and extend it from comparing models across corpora to comparing arms across social strata. Per-stratum performance differences between arms are partially pooled across strata and tasks. This borrows strength for small intersectional groups, which is the low-data problem that blocks RecZilla-style meta-learners needing millions of prior evaluations (McElfresh et al.).

For each stratum $s$, the rule selects the arm $a$ that minimizes posterior expected calibration error. It considers only arms whose population AUROC lies within a region of practical equivalence (we set 0.01, and also report 0.005 and 0.02) of the best arm, with probability at least 0.9. When no arm is credibly better within a stratum, the rule falls back to the simplest arm. We do not use the contextual bandit of PAK-UCB, because readmission and 1-year mortality rewards arrive weeks to a year late. We do not use label-free routing such as Smoothie either, because retrospective labels exist and our arms share base learners, which violates its assumption that errors are independent.

The analysis tool is a simple decomposition of each arm's calibration gain over Arm 1. The first part is the share that disappears once Arm 1 is recalibrated per stratum (an offset). The second is the share that remains together with a within-stratum AUROC gain (a ranking change). This decomposition is the deliverable that tells practitioners whether collecting individual social data is worth the cost.

Baselines

The do-nothing reference is Arm 1 chosen by pooled AUROC, which is what standard pipelines deploy. Beating it means lower worst-stratum ECE and a smaller worst-stratum false-positive gap at AUROC within the equivalence region.

The strongest existing method is the pooled social-feature-enriched model of Yang et al.. The file reports an in-hospital mortality ECE of 0.03 for it, down from 0.11 clinical-only. It reports no significant population AUROC or AUPRC change, and the file gives no absolute AUROC values. We re-implement it on the same linked data, splits, and hyperparameter search budget as every arm, using equal random-search trials per learner. Beating it means a lower worst-stratum ECE, since the 0.03 is a population figure, with no stratum made worse beyond the equivalence region.

The cheapest serious competitor is Arm 2 deployed globally, meaning stratum-wise recalibration of a clinical model. No reported result was found on Lacuna for stratum-recalibrated clinical models on MIMIC-IV.

Static decoupling with tuned thresholds follows Hu et al.. On a different, private cohort they equalized false-negative rate at 25% using thresholds of 0.296 and 0.245, starting from default-threshold false-positive rates of 27.71% vs 5.92%. We run it on our strata with thresholds tuned on validation data only.

Two ablations isolate the key ingredients. The first replaces partial pooling with independent per-stratum selection. The second replaces it with a single global selection made using our calibration-under-equivalence criterion. That second ablation separates the value of a better objective from the value of selecting by context.

Why not just…?

Why not just recalibrate the clinical model within each social stratum? This is the real threat, and the proposal is built around it. It is Arm 2. If stratum-recalibrated Arm 1 matches the enriched models on worst-stratum ECE and false-positive gap everywhere, our decomposition will report that social features act as a grouping variable. The selector will then pick Arm 2 everywhere, and the actionable conclusion is that hospitals need stratum labels, not richer social models. The experiment settles the question either way.

Why not just switch the global selection criterion to calibration and keep one model? The insurance-dependent error flip in Chen et al. suggests the best arm differs by setting, but that was shown with a single logistic model, not across arms. The global-criterion ablation tests this directly. If it matches context-conditioned selection on held-out data, the per-context machinery is unnecessary and we will say so.

Why not just split by race and tune thresholds? Hu et al. report that race-specific models were insufficient without threshold tuning. The disparities reported by Yang et al. are intersectional, which coarse demographic splits miss. We include this approach as Arm 5 rather than dismissing it, so it competes on the same strata.

Proposed Experiments

The primary data are MIMIC-IV linked to the three community databases as in Yang et al.. We hold out the latest admission-year group as the test set, so that selection is made on earlier data and evaluated on later data. The note-derived arm uses MIMIC discharge summaries with the MIMIC-SBDH-trained extractor. External transfer is checked on the All of Us cohort, where Yang et al. validated their findings. A selection decision fit on MIMIC-IV is applied there without refitting. Metrics are population AUROC and AUPRC, per-stratum and worst-stratum ECE, per-stratum false-positive and false-negative rates, and within-stratum AUROC. All are reported as posterior intervals.

Three checks decide the outcome. First, the offset-versus-ranking decomposition per task and stratum. The prediction is that ranking gains from Arm 4 concentrate in the highest-vulnerability quartile, and the hypothesis fails if within-stratum AUROC gains are absent everywhere, including after the noise perturbation. Second, context-conditioned selection must beat both the pooled enriched model and global Arm 2 on worst-stratum ECE in the temporal hold-out. Third, the selected per-context arms must keep that advantage on All of Us. If selection is context-dependent only within one hospital, the Regularity Effect implies decisions must be refit per site, and we would report that as the main finding.