Uncertainty That Follows the Chart: Conformal Calibration by Modality Availability

When a hospital model is unsure about a patient, the reason is often mundane: the record is thin. The patient has no prior visits in the system, no chest X-ray, or a two-line triage note. This proposal tests a simple hypothesis. In multimodal clinical prediction, much of the demographic gap in how reliable a model is (not just how accurate) is caused by gaps in what data exist for each patient. If that is true, then calibrating uncertainty separately for each pattern of available data should do three things at once. It should even out reliability across demographic groups without using demographics at prediction time. It should keep uncertainty estimates narrow enough to act on. And it should hand the clinician a readable reason for a wide estimate, such as "no imaging, fewer than two prior encounters."

The tool here is conformal prediction, a wrapper around any trained model. It uses a held-out calibration set to output a set of plausible labels that contains the true label with a chosen probability, say 90%. For a binary task like in-hospital mortality, the set is either one label or both. "Both" amounts to the model abstaining, so the share of patients who receive both labels measures how often the model is unhelpful. The standard fair version, group-conditional ("Mondrian") conformal prediction, calibrates separately for each demographic group. We propose instead to calibrate over a shallow, human-readable partition of patients by their information profile: which modalities are present, how long the structured history is, and how much text exists. We then measure how much of the demographic reliability gap this closes.

Core Claim

On public multimodal clinical benchmarks, we claim that conformal calibration conditioned on an interpretable information-profile partition will do two things relative to demographic Mondrian calibration. First, it will match demographic Mondrian on worst-group coverage of true positives, which is the coverage that matters for underdiagnosis. Second, it will produce smaller gaps between groups in set size and abstention. As a by-product, it will give a per-patient explanation of uncertainty that is checkable: adding the missing modality should, on average, actually narrow the set.

The claim is falsifiable at its root. If conditioning on information profile removes less than about half of the worst-group coverage gap left by marginal calibration, then the demographic gap is not mainly a data-availability gap. In that case the method reduces to a weaker Mondrian, and we will report it as such.

Why This Gap, and What It Builds On

Three findings motivate the hypothesis. Autoregressive EHR Foundation Models with Multimodal Inputs adds notes and chest X-rays to a structured-EHR model on MIMIC-IV and finds "modality competition," where the dense structured stream crowds out sparse ones. Zero-shot ICU mortality AUROC rose from 0.8651 to 0.8751 with notes and then fell to 0.8624 when X-rays were added. MC-BEC finds the same effect in emergency care, where labs and radiology lowered decompensation AUPRC. It also finds that younger patients are disproportionately missed. A Fairness Audit of Medical Imaging Foundation Models shows on INSPECT that age dominates bias, with near-chance AUROC (~0.508) for ages 18–40 in two foundation models.

None of these papers produces uncertainty estimates. None asks whether the patients who are underserved are also the patients with sparse records, for whom the suppressed modalities carry the signal. Benchmarking Foundation Models with Multimodal Public Electronic Health Records reports that fusion on MIMIC-IV did not worsen demographic parity or equalized odds. Its result constrains us: point-prediction fairness may look fine while reliability differs, which is the concern raised in Demystifying Algorithmic Fairness in an Uncertain World.

Multimodal conformal work exists but is calibrated only marginally. THCM-CAL builds patient-specific causal graphs over notes and codes and then applies split conformal prediction to ICD sets. Probing, Fusion, and Trustworthiness fuses histopathology and transcriptomics and shows that conformal sets recover 79.7% of point errors. Neither looks at subgroups. We reuse their recipe of split conformal prediction on top of a fused model and add the subgroup question.

Fair conformal methods supply our machinery and our main constraints. Beyond Procedure: Substantive Fairness in Conformal Prediction shows that demographic Mondrian inflates set-size gaps and harms downstream decisions; on BiosBias its fairness score maxROR is 8.1% versus 1.6% for its Label-Clustered variant, which groups labels by difficulty. We share its goal of substantive fairness, but our partition is over patients' data availability rather than labels. Equal Opportunity of Coverage in Fair Regression proves that equal coverage across groups and equal coverage conditional on the outcome cannot both hold when outcome rates differ by group. Mortality differs by age, so we target coverage conditional on the true label within each cell, and we report both quantities rather than pretending the conflict away. Identifying Homogeneous and Interpretable Groups for Conformal Prediction supplies the partition learner, MCR_DTREE. It trains shallow decision trees against an adversarial judge that searches for the worst-miscovered group, but it was tested only on tabular UCI regression. Our change is to restrict its split variables to information-profile descriptors and to apply it to classification on real clinical data. The ablations test whether that restriction matters.

Research Program

The base predictors are fused multimodal models trained under the MIMIC-IV pipeline of Benchmarking Foundation Models. That pipeline combines demographics, vitals time-series, chest X-rays, and radiology reports for mortality and length of stay greater than 3 days. We add MC-BEC for decompensation and disposition, and INSPECT for pulmonary embolism diagnosis and 1-month mortality. Demographics are excluded from the profile descriptors and used only for auditing.

For each patient we compute a small descriptor vector. It records the presence of each modality, the count of prior encounters, the number of structured events in the observation window, note token length, and time since the last imaging study. We also add one model-derived quantity: how much the nonconformity score changes when each present modality is masked. This captures modality reliance, the patient-level face of modality competition.

MCR_DTREE then learns a tree of depth at most 3 over these descriptors, with a minimum leaf size so that each cell keeps at least a few hundred calibration patients. Larger cells avoid the sample starvation that intersectional demographic Mondrian suffers (FedCF). Within each leaf we run label-conditional split conformal prediction.

The explanation shown to a clinician is the leaf's rule, together with the masked-modality score changes. We will not present this as causal. Instead we test its fidelity directly. For patients who have a modality, we mask it and check whether the model's set widens as the rule predicts. The tree is fit on one split, calibrated on a second, and audited on a third, so the coverage guarantee within each leaf remains valid.

The second analysis is a mediation audit. It asks how much of the gap between demographic groups under marginal calibration disappears once we condition on the leaf. We use a reweighting decomposition: reweight each group's patients to match the reference group's leaf distribution and recompute coverage and abstention.

Baselines

The naive reference is marginal split conformal prediction on the same fused model. This is what THCM-CAL and the cancer-fusion study do. It shows the size of the demographic reliability gap before any intervention.

The strongest direct competitor is demographic Mondrian with label-conditional calibration, applied to sex, race, age band, and their intersections where cells permit. It is the method a reviewer would expect, it achieves exact coverage within groups by construction, and it uses the protected attribute the proposal deliberately withholds. Beating it means matching its worst-group positive-label coverage within 2 points while showing a smaller cross-group gap in abstention rate and a smaller maxROR, computed as in Beyond Procedure. The only comparable figure on Lacuna is that paper's BiosBias result (Mondrian 8.1% vs Label-Clustered 1.6% maxROR). We found no reported conformal coverage or set-size result on Lacuna for MIMIC-IV mortality, MC-BEC, or INSPECT, so we will establish all calibration baselines ourselves. For the underlying predictor, the reference point is mortality AUROC 0.8918 (Text-Embedding-3-Large + Swin) from Benchmarking Foundation Models. Our fused model must come close to this number, or the calibration comparison is uninformative.

Label-Clustered CP is run as published, as the best existing method for substantive fairness. Unrestricted MCR_DTREE is given all tabular features plus PCA-reduced embeddings, with the same depth limit, leaf size, and calibration split. It is the ablation that isolates our key ingredient, the restriction to information-profile descriptors. A random partition with the same number of leaves controls for the effect of simply having more cells.

All methods share the same fused model, the same three-way split, five random seeds, and the same hyperparameter search budget for any partition learner. On INSPECT we also apply calibration on top of the age-targeted adversarially debiased model from the fairness audit. That model cut the underdiagnosis-rate gap from 0.333 to 0.069 at a cost of 0.011 AUROC, and we ask whether calibration still adds anything after debiasing.

Why not just…?

Why not just use demographic Mondrian conformal prediction? It guarantees coverage within each group, but only by widening sets for the groups the model serves worst. Beyond Procedure shows this raises downstream unfairness. On MIMIC-IV, intersectional cells for minority groups will be small. Mondrian is our main baseline. If it wins on both coverage and abstention gap, the proposal fails, and we will say so.

Why not just fix modality competition during training, for instance with modality dropout or gated cross-attention as in Autoregressive EHR Foundation Models? This is a real threat, because removing the cause could make calibration over profiles unnecessary. We therefore train a modality-dropout version of every fused model and repeat the mediation audit on it. If the profile-explained share of the demographic gap collapses under dropout training, the right recommendation is a training fix. Profile calibration would then be useful mainly as a diagnostic, and the paper reports that outcome.

Why not just run MCR_DTREE on all features? It may find the same cells or better ones. We prefer profile descriptors because they are legible and actionable: "order imaging" is an action, while "embedding dimension 14 > 0.3" is not. That preference is only justified if accuracy is not sacrificed. The unrestricted-MCR_DTREE ablation settles this question. A tie on coverage and set-size metrics would still favor the restricted version on interpretability, but we will report the tie plainly.

Why not just equalize error rates post hoc? Equalized-odds thresholding in Exploring Text Specific and Blackbox Fairness Algorithms cost substantial AUC, and equal error rates can hide unequal confidence. We include it as an audit row so that its group abstention and coverage behavior are visible next to the conformal methods.

Proposed Experiments

The first decisive check is the mediation audit on MIMIC-IV mortality and INSPECT. Under marginal calibration, we compute the worst-group gap in coverage of positive labels across age, race, and sex. We then compute the share removed by reweighting to the leaf distribution. Above one half supports the hypothesis; below one quarter refutes it.

The second check is the head-to-head comparison across all three benchmarks. The metrics are worst-group label-conditional coverage at 90%, the cross-group abstention-rate gap, and maxROR. We also report the conflict between equal coverage and outcome-conditional coverage implied by the mutual exclusivity result, as the two numbers side by side.

The third check is explanation fidelity. For held-out patients who have a modality, we mask it and measure whether the set widens in the direction and roughly the magnitude implied by the leaf rule. We report the agreement rate against a random-rule control.

The fourth check is robustness: everything is repeated with the modality-dropout fused models and with the debiased INSPECT model. This tells us whether profile calibration still adds value once the training-side fixes are applied.