Calibrating Clinical Risk Along the Deprivation Gradient

Clinical risk models are most likely to be wrong about patients whose lives outside the hospital are hardest. Area-level social measures, such as the Social Vulnerability Index or county health rankings linked through a patient's zip code, barely change how well these models rank patients. They do change how trustworthy the predicted probabilities are. On MIMIC-IV linked to three community databases, Yang et al. found that adding social determinants of health (SDoH, the non-clinical conditions such as income, housing, and transport that shape health) left AUROC and AUPRC flat. The same addition cut the expected calibration error (ECE, the average gap between predicted risk and observed frequency) of in-hospital mortality from 0.11 to 0.03 and narrowed false-positive-rate gaps for vulnerable subgroups. The value of social context therefore sits in calibration, not discrimination. That points to a tool built for calibration: focal loss, a training objective that down-weights examples the model already gets right. Mukhoti et al. showed it acts as an implicit penalty on overconfidence, and AdaFocal showed its strength should vary across the model's confidence levels.

We propose to make the focal strength adapt jointly to confidence and to a patient's position on a continuous neighborhood-deprivation scale. The model is then pushed during training to be calibrated within each deprivation stratum, not only on average. The question is narrow and testable. Can a deprivation-indexed focal objective give calibration that holds across deprivation levels and survives temporal and cohort shift, where cheaper post-hoc fixes break? The fixes in question are per-group temperature scaling and per-group thresholds.

Core Claim

Calibration error in SDoH-enriched clinical models is not uniform. It concentrates in particular combinations of confidence and deprivation. A model can be overconfident about low-risk patients from affluent areas and underconfident about high-risk patients from deprived ones. A single global correction cannot remove both errors at once. AdaFocal already adapts the focusing parameter per confidence bin, but its bins ignore who the patient is. Fairness corrections such as Hu et al.'s race-specific thresholds work along protected attributes. The structural-injustice position paper argues those attribute-based corrections can worsen outcomes because they ignore heterogeneity within groups along the deprivation gradient.

Our claim is that indexing the focal parameter by confidence bin and deprivation stratum during training will do two things. It will reduce worst-stratum calibration error relative to both confidence-only adaptive focal loss and plain cross-entropy, without losing AUROC. It will also keep that advantage under temporal shift better than post-hoc per-stratum recalibration.

Mechanism

Let $p_t$ be the model's probability for the true label, $b$ the confidence bin a sample falls into, and $d$ its deprivation stratum. Strata are deciles of a continuous area index, never race. The per-sample loss is

$$\ell = -(1-p_t)^{\gamma_{b,d}} \log p_t ,$$

and after each epoch every focal parameter is updated multiplicatively from validation statistics:

$$\gamma_{b,d} \leftarrow \gamma_{b,d}\, \exp\big(\lambda\,(\bar{C}_{b,d} - \bar{A}_{b,d})\big).$$

Here $\bar{C}_{b,d}$ and $\bar{A}_{b,d}$ are the mean confidence and the observed accuracy of validation samples in that cell, and $\lambda$ is a step size. This is AdaFocal's update rule. The new element is the second index.

When a cell is underconfident, we follow AdaFocal and switch to the inverse focal form, which up-weights easy samples. We take this from AdaFocal's method and from Dual Focal Loss's finding that static focal loss tends to push models toward underconfidence. We also cap $\gamma$ for rare outcomes. Shah et al. proved that under extreme imbalance with large $\gamma$, focal loss enters an over-suppression regime where small minority probabilities shrink further. One-year mortality, and especially readmission, sit closer to that regime than CIFAR does.

Two engineering problems are specific to this setting. The first is that a full confidence × decile grid leaves many cells nearly empty, which is AdaFocal's known validation-overhead weakness made worse. We handle this by estimating cell statistics with a kernel smoother over the continuous deprivation index, so adjacent deciles share strength. The second is that the strongest tabular baselines are gradient-boosted trees. We will implement the loss as a custom XGBoost objective, supplying its gradient and Hessian, and also train a small multilayer perceptron. That way any effect is not an artifact of model class.

Research Program

The program has three parts.

The first part is descriptive. On MIMIC-IV-SDOH we will map where calibration error lives for cross-entropy models, with and without SDoH features, across the confidence × deprivation grid. If the error is not concentrated in a few cells, the joint indexing has nothing to fix. We would then report that as the main finding and stop.

The second part trains the deprivation-indexed focal objective. We compare it against the baselines below on Yang et al.'s three tasks: in-hospital mortality, 30-day readmission, and one-year mortality.

The third part is the stress test that justifies loss-level work at all. Models are fit on earlier admission years of MIMIC-IV and evaluated on later ones. Where data access allows, we also evaluate on the All of Us cohort, which Yang et al. used for validation. Post-hoc calibrators are fit only on the earlier period. This tests whether calibration learned inside the model transfers better than calibration fitted on top of it. Mukhoti et al. report this pattern for focal loss under image corruption and domain shift, but it has not been shown for subgroup calibration in clinical tabular data.

Baselines

The naive reference point is an XGBoost model trained with standard log-loss on clinical features only. Next to it sits the same model with Yang et al.'s zip-code-linked SDoH features. That second model is the strongest existing result on this question: Yang et al. report in-hospital mortality ECE of 0.03 for clinical plus SDoH features, down from 0.11 for clinical only. We will reproduce both on our splits rather than borrow the numbers, because our evaluation splits differ.

For the main comparison, beating this baseline means two things. Overall ECE must stay at or below the reproduced value. Worst-decile ECE must be significantly lower under bootstrap confidence intervals. AUROC must stay within the bootstrap interval of the baseline.

The strongest post-hoc competitor is temperature scaling fit separately within each deprivation decile on the validation set. It is cheap, it targets exactly the subgroup calibration we care about, and in-distribution it may match us. No reported result for it was found on Lacuna for MIMIC-IV-SDOH.

The strongest training-time competitors are static focal loss and confidence-only AdaFocal, the latter being the method we extend. For method context, AdaFocal reports CIFAR-100 ResNet-50 ECE of 1.36% against 5.64% for focal loss and 17.17% for cross-entropy. Dual Focal Loss reports 1.08%. No reported result was found on Lacuna for either method on clinical SDoH data. Dual Focal Loss is included as a second adaptive baseline.

On the fairness side we include two more baselines. One is Hu et al.'s post-hoc threshold equalization. They equalized false-negative rate at 25% using thresholds of 0.296 and 0.245, on a cohort where XGBoost reached 74.49% AUC. The other is distributionally robust optimization over deprivation strata, which Hu et al. also evaluate.

The ablation that isolates our key ingredient replaces deprivation strata in the grid with three alternatives: random strata of equal size, race strata, and confidence-only bins. If random strata do as well as deprivation strata, the gain comes from finer binning, not from social structure. A second ablation removes the inverse-focal switch and the rare-outcome cap.

Every method gets the same splits, the same feature sets, and the same 50-trial hyperparameter search per method over learning rate, depth, and loss parameters. Every method also gets the same validation data for its calibration statistics or post-hoc fitting.

Why not just…?

Why not just add the SDoH features, as Yang et al. did? That is the bar to clear, not an alternative to reject. The features already deliver most of the average calibration gain. What is unknown is whether the residual error is concentrated in deprived strata and whether it persists under shift. The descriptive first stage answers this directly. We run every loss with and without SDoH features. If SDoH features alone leave worst-decile ECE close to overall ECE, our method has no room to help, and we will say so.

Why not just apply per-decile temperature scaling after training? This is the real threat. Within a single time period, a calibrator fit to each stratum may match any training-time method. Our case rests on two weaknesses. Post-hoc scaling is global within a stratum, so it cannot fix opposite errors at different confidence levels, which is AdaFocal's argument. It is also fit to one time window, and Mukhoti et al. found that temperature-scaled cross-entropy models lose calibration under shift while focal-trained models hold it better. The temporal-transfer experiment settles this. If per-decile temperature scaling matches us on later admission years, the method adds nothing worth its complexity.

Why not just shift decision thresholds per group? Hu et al. show that threshold shifting equalizes one error rate. It leaves the probabilities themselves miscalibrated, and those probabilities are what clinicians read. When it is done along race, the position paper argues it can harm deprived members of advantaged groups. We report threshold equalization on false-negative rate and also its calibration within deprivation strata, which we expect to stay poor.

Why not just class-weighted cross-entropy or group robust optimization? Fixed inverse-frequency weights change the effective base rate and so miscalibrate by design. Robust optimization targets worst-group loss, not calibration. Both are in the baseline set. If either matches our worst-decile ECE, the confidence-indexed part of our loss is unnecessary.

Decisive Checks

The proposal succeeds if three things hold for at least two of the three MIMIC-IV-SDOH tasks. First, deprivation-indexed focal loss lowers worst-decile ECE relative to both reproduced SDoH-feature cross-entropy and confidence-only AdaFocal, with no AUROC loss. Second, deprivation strata beat random and race strata in the ablation. Third, on later admission years the method's worst-decile ECE stays below that of per-decile temperature scaling fit on earlier years.

We will also report calibration slope and per-decile false-positive and false-negative rates. We will report race gaps inside each deprivation decile to test the position paper's within-group heterogeneity concern directly. We will report readmission separately, since at its prevalence the over-suppression cap is most likely to matter.

A clean negative result on the third check would still be useful. It would tell practitioners that subgroup calibration in this setting is a post-hoc problem and that the loss can be left alone.