Adding neighborhood-level social and economic data to clinical prediction models is now a common recommendation. The strongest evidence for it is a specific claim. Zip-code-linked social indicators do not make a hospital mortality model more accurate overall, but they make its predicted probabilities more trustworthy for disadvantaged groups. That benefit is also reported to carry over to a second, nationwide cohort. This note proposes to test whether that claim survives the many ordinary data-preparation choices any analyst must make along the way: how to fill in missing neighborhood scores, which social index to link, whether to fit feature scaling on training data alone, and even how the calibration metric bins its predictions. In NLP, such choices have been shown to flip model rankings.
The approach treats preprocessing choices the way a trial treats a drug. Because pipeline choices can be randomized, we can cross them in a full factorial design. The design measures how much of the reported benefit is caused by the social data itself and how much by the pipeline, both on the development hospital and on an external cohort. If the benefit holds across the defensible pipeline space and transfers externally, the field gains a robust result. If it does not, we learn which specific choice manufactures it, and whether simpler fixes such as local recalibration or a missingness flag reproduce it.
Core Claim
Evaluating the Impact of Social Determinants on Health Prediction built MIMIC-IV-SDOH by linking MIMIC-IV patients through 5-digit zip codes to three public databases: County Health Rankings, the CDC Social Vulnerability Index, and the AHRQ SDOH database. It found these features "accuracy-neutral" but "fairness-active." In-hospital mortality expected calibration error (ECE, the average gap between predicted and observed risk across probability bins) fell from 0.11 to 0.03, and the pattern was confirmed on the All of Us cohort. That paper explicitly leaves open how imputation, scaling and similar choices affect these calibration and fairness metrics.
Our claim is testable and could fail. We hypothesize that the calibration gain from social features varies across reasonable pipelines by an amount comparable to the gain itself. We further hypothesize that a large share of the gain is carried by the missingness pattern of the linked indices rather than their social content. Missing or unlinkable zip codes are plausibly correlated with insurance, housing instability and race.
If the gain is stable across pipelines, survives on the external cohort, and disappears when the social values are shuffled while their missingness is kept, the original claim is strengthened. If it collapses under a few choices, we will have identified the "Strict versus Boundaries" of clinical fairness evaluation.
Why this intersection, and what each line of work contributes
Preprocessing inconsistency supplies the core mechanism and motivating evidence. Let's Stop Incorrect Comparisons in End-to-end Relation Extraction! showed that one BERT model scores 59.6 under Strict matching and 62.9 under Boundaries matching on ACE05, and that circulating versions of the dataset range from 10k to 14k sentences. Aditya Joshi et al. showed that changing one stemming or stop-word setting inverts ROUGE rankings. Rescaling and other forms of unsupervised preprocessing introduce bias into cross-validation showed that fitting unsupervised steps before splitting biases internal estimates. None of these studied clinical calibration, fairness, or external cohorts. We carry their logic into a setting where the stakes are clinical thresholds.
Social determinants supply the application regime. Area-level indices are known to blur household conditions (the ecological concern raised by Yang et al.). Causal Fairness Assessment of Treatment Allocation with Electronic Health Records found that CABG allocation disparities persisted whether or not social variables were included, and reported that race and social variables are missing not at random in structured records. That paper constrains our interpretation: rich clinical features may already absorb social disadvantage, which leaves room for the social features' apparent value to come from somewhere else, such as their missingness. We deliberately stay with structured, area-level indices. MIMIC-SBDH shows that text-extracted social factors add their own failure layer (44.5% negation-test failures for Bio-ClinicalBERT on Community-Present). Mixing that layer in would make the attribution question unanswerable.
Confounder control enters in two places. It is the lens we use to say why published comparisons are untrustworthy: in a typical paper, "with social features" and "without" are not just two feature sets but often two pipelines, so pipeline choice is confounded with the comparison. It is also the remedy: a fully crossed design removes that confounding by construction.
External cohort validation is the evaluation setting. Wang et al. ran one harmonized OMOP pipeline across 11 databases and still saw external AUC fall to 0.50 on a German outpatient database. Darabi et al. saw AUC drop from 0.912 on eICU to 0.855 on MIMIC-III without separating pipeline effects from population effects. Both treat preprocessing as fixed. All models are local argues that transportability is the wrong target and that local recalibration should replace it. We take that argument seriously as a competitor rather than a backdrop. Partial Transportability for Domain Generalization warns that adjusting for non-transportable variables can hurt target performance. It requires a known selection diagram, which we do not assume. We use it only as a reason to expect that social features tied to one city's geography may not transfer.
Research Program
We define a pipeline $p$ as one setting of each factor in a crossed grid. The factors are:
- Social data source: County Health Rankings, Social Vulnerability Index, AHRQ SDOH, or all three.
- Missing-index handling: complete-case, median fill, median fill with an explicit missingness indicator, or iterative multivariate imputation.
- Scope of scaling and imputation fits: training split only, or pooled.
- Clinical feature window: first 24 or 48 hours.
- Outlier clipping: on or off.
- Model family: logistic regression or XGBoost, the families used by Yang et al.
- Calibration metric binning: equal-width versus equal-mass bins, at two bin counts.
The binning factor is the clinical analogue of Strict versus Boundaries. It changes the metric without changing any model.
For each pipeline, the quantity of interest is the social-feature effect
$$\Delta(p, g, c) = \mathrm{ECE}_{\text{clin}}(p, g, c) - \mathrm{ECE}_{\text{clin+SDoH}}(p, g, c),$$
where $g$ is a patient subgroup (race, sex, insurance or a deprivation tercile) and $c$ is the cohort: MIMIC-IV-SDOH internal test, or All of Us external, using the same linkage as Yang et al. We compute the same difference for AUROC, AUPRC and false-positive-rate gaps.
Because pipelines are assigned by design, a standard variance decomposition of $\Delta$ across factors attributes its spread to each choice and their interactions without confounding. We report it separately for the internal and external cohorts. We compare the pipeline-induced spread against seed-and-bootstrap spread at a fixed pipeline. This tells us whether analyst choices matter more than sampling noise.
Two ablations isolate what the social features actually carry:
- Indicator-only: replace every social feature with only its missingness indicator.
- Permuted-content: randomly permute social values among patients with observed values, keeping each patient's missingness unchanged.
If the permuted-content model retains most of the external calibration gain, the "fairness-active" effect is a data-availability signal, not a social one.
Baselines
The do-nothing reference is the clinical-only model under the single pipeline described by Yang et al. It defines how miscalibrated the subgroups are before anything is added.
The strongest existing method is that paper's own clinical-plus-social model, which reports in-hospital mortality ECE falling from 0.11 to 0.03. For AUROC and AUPRC no number is reported on Lacuna, only "accuracy-neutral." For external All of Us performance, no numeric result is reported on Lacuna for Yang et al. We first reproduce their pipeline as closely as the description allows and treat our reproduced numbers as the reference. Any failure to reach roughly 0.03 is itself reported as a finding about under-specified preprocessing.
Local recalibration is the competitor suggested by All models are local. We apply Platt or isotonic recalibration, both overall and per subgroup, to the clinical-only model, fitted on a small held-out slice of each target cohort. We sweep slice sizes so the comparison reflects realistic local data budgets.
The two ablations above, indicator-only and permuted-content, isolate the key ingredient: real social content.
Fairness rules apply to all comparisons:
- Every arm uses identical patient splits, the same external cohort, and the same hyperparameter search budget per pipeline cell, a fixed small random search.
- All preprocessing is fit on training data except in the cells that deliberately test pooled fitting.
We will say the original claim is robust if two conditions hold. First, the real social model's subgroup ECE gain exceeds both recalibration of the clinical model and the permuted-content ablation across at least the large majority of pipelines, with bootstrap intervals excluding zero, on both cohorts. Second, the factor decomposition attributes less of $\Delta$'s variance to any single choice than to the social-versus-clinical contrast itself.
Why not just…?
Why not just run everything on a common data model so preprocessing is fixed? A common schema fixes codes, not choices such as imputation, linkage source, windows or metric binning, and none of those live in OMOP tables. Wang et al. ran one harmonized pipeline and still saw external AUC of 0.50 at one site. A single canonical pipeline would also answer only whether the effect holds under that pipeline. Our question is whether the effect depends on which pipeline was picked.
Why not just recalibrate locally and stop worrying about social features? This is the most serious threat, and we treat it as a possible outcome rather than a strawman. Local recalibration directly targets ECE, and per-subgroup recalibration directly targets subgroup ECE. If a few hundred local patients suffice to match the social model's gain, the practical case for linking social indices to improve calibration weakens considerably. Their value would then rest on interpretability or settings without local labels. The slice-size sweep tells us the exact data budget at which recalibration catches up, which is the number a deploying hospital needs.
Why not just report bootstrap confidence intervals on the one published pipeline? Bootstrap intervals capture sampling noise under a fixed analysis, not systematic analyst choices. The NLP evidence (Taillé et al., Joshi et al.) shows that systematic choices can exceed model differences. Our design measures both spreads side by side. If pipeline spread is no larger than seed spread, we will say so, and the robustness concern is put to rest.
Why not just add a missingness indicator and call it a social feature? We test this directly. If indicator-only matches the full model, that is the main result: the fairness benefit is real but is a signal about data availability. The clinical and policy reading of that is different from "neighborhood deprivation explains miscalibration."
Proposed Experiments
The primary experiment is the full factorial on in-hospital mortality, trained on MIMIC-IV-SDOH and evaluated on its internal test split and on All of Us. The outcomes are subgroup ECE, AUROC, AUPRC and false-positive-rate gaps, plus the variance decomposition of $\Delta$. A secondary experiment repeats the grid for 30-day readmission and one-year mortality. In those tasks acute physiology dominates less, so social features have more room to matter, and we expect the content-versus-missingness contrast to be sharper there.
A tertiary check applies the clinical-only part of the grid (windows, imputation, clipping, scaling scope) to the eICU-to-MIMIC-III transfer of Darabi et al.. Their reported external AUC is 0.855 versus 0.741 for logistic regression. The check asks how much of their 0.057 internal-to-external drop moves with pipeline choice alone.
Three results are decisive:
- Whether the sign of the subgroup ECE gain flips in any non-trivial fraction of pipelines.
- Whether the binning factor alone moves ECE by an amount comparable to 0.08, the published 0.11-to-0.03 gap.
- Whether the permuted-content model retains the external gain on All of Us.
Each outcome changes what a clinical team should do: trust the linkage, recalibrate locally, or report pipelines alongside fairness numbers.
References
- Evaluating the Impact of Social Determinants on Health Prediction
- Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!
- Red-faced ROUGE: Examining the Suitability of ROUGE for Opinion Summary Evaluation
- Rescaling and other forms of unsupervised preprocessing introduce bias into cross-validation
- Causal fairness assessment of treatment allocation with electronic health records
- MIMIC-SBDH: A Dataset for Social and Behavioral Determinants of Health
- Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network
- A Clinically Practical and Interpretable Deep Model for ICU Mortality Prediction with External Validation.
- All models are local: time to replace external validation with recurrent local validation
- Partial Transportability for Domain Generalization