When a clinical risk model sees a patient record with missing lab values, a hidden step fills those gaps before the model runs. Multiple imputation is a statistical method that fills each gap many times with different plausible values, to reflect that we do not know the true one. The model's output then depends partly on guesses, and the person reading it never sees this. This proposal tests whether an interface that shows the spread across those fill-ins helps people rely on the AI more wisely. The display would read, for example, "across 20 plausible fill-ins of this patient's missing lactate, the model called high risk in 13." The comparison is against a single confidence number.
The bet is narrow and falsifiable. Pooled confidence is calibrated on average across all patients, so it can look equally trustworthy for a fully measured record and for one where the deciding variable was never measured. Disagreement across imputations is a per-record signal that such averaging washes out. It is also something people may be able to act on: they can discount the prediction, or order the missing test. The AI literacy literature argues that users need a working model of how data drives predictions. Here, missing data is the concrete, teachable instance of that idea.
Core Claim
We claim two things. First, for tabular clinical prediction under real missingness, the rate at which imputations disagree on a record's predicted label predicts whether the AI is wrong, beyond what the pooled calibrated probability already predicts. Second, if that holds, presenting the disagreement in natural-frequency form reduces people's over-reliance on wrong predictions, and the reduction concentrates on heavily imputed records. The first claim is an offline check that gates the second, which is a human-subject experiment. If the offline signal is absent, we stop. In that case the right answer is simply to show calibrated confidence, and we will report it as such.
What Prior Work Establishes
The motivating evidence comes from two papers that never meet. Designing for Appropriate Reliance tested four uncertainty formats on melanoma image classification, a setting with complete inputs and 50 participants. Moving from raw to calibrated percentages did not change behavior. Over-reliance stayed at 44–61% across all formats. Only a calibrated-frequency statement ("in 100 samples like this…") tied the AI's confidence to users' switching ($\chi^2 = 26.05$, $p < .001$). The human-AI team reached 67% accuracy against the AI's 80%. That paper supplies our presentation format, but it has no notion of uncertainty that originates in the inputs.
Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction models a variance for every imputed entry on MIMIC-III (about 94% missing) and PhysioNet 2012 (about 81% missing). It shows that this variance tracks the actual imputation error and uses it to down-weight uncertain inputs inside the network. Its authors name visualization for clinicians as future work. We take up that invitation. We use a model-agnostic signal, disagreement across multiple-imputation draws, rather than an architecture-internal gate, and we put it in front of people.
Two papers constrain the design. Feature Importance Metrics in the Presence of Missing Data shows that imputing with conditional means leaks predictive signal into the filled-in values and distorts explanations. We therefore use stochastic draws from chained-equations imputation, never a single best guess. Correlation visualization under missing values shows that an imputation method can have low value-level error while compressing correlations, which it calls the "RMSE trap." We therefore judge the imputation layer by whether its disagreement predicts downstream errors, not by reconstruction error.
On the literacy side, What do medical students actually need to know about artificial intelligence? argues that clinicians must act as gatekeepers who understand how data shapes outputs. Making Sense of Machine Learning finds that deliberately broken "failure artifacts" teach well, and notes that pedagogy for missing-data systems is unaddressed. We borrow the failure-artifact idea for a short training arm, and we treat prior literacy as a measured moderator rather than the headline.
Research Program
The imputation layer uses chained equations (MICE), which fill each incomplete variable by regressing it on the others and drawing from the resulting predictive distribution. We generate $M = 20$ completed datasets per record. A gradient-boosted classifier is trained once on imputed training data. At test time we score each of a record's $M$ completions. The pooled probability $\bar p$ is the mean across completions, calibrated on held-out data. The disagreement rate is $d = \min(k, M-k)/M$, where $k$ is the number of completions whose predicted label is positive. RefiDiff (RefiDiff), a diffusion imputer warm-started by a boosted model, serves as a second imputer, to check that findings do not hinge on MICE.
The offline gate works as follows. On PhysioNet 2012 mortality, using first-48-hour summary features so that records are static tables a human can read, we fit a logistic model of AI error on the calibrated $\bar p$ alone. We then fit it on $\bar p$ plus $d$ plus the fraction of features imputed. The gate passes if adding $d$ improves held-out error prediction (log-loss and AUROC for predicting errors), with a bootstrap interval excluding zero. We also require the effect to hold within bins of $\bar p$. We replicate on the mixed-type Adult and Default tables from the RefiDiff benchmark, under synthetic MAR and MNAR masks, where the true values of the masked entries are known.
The human study is preregistered and between-subjects. Participants see 40 patient summaries, give an initial mortality judgment, then see the AI's prediction and may revise it. Records are sampled so that half are heavily imputed and so that AI errors appear at the model's natural rate within each stratum. Arms are defined by what accompanies the AI's prediction:
- no uncertainty at all;
- calibrated frequency for $\bar p$ only;
- calibrated frequency plus a list of which features were imputed, with no spread shown;
- calibrated frequency plus the fill-in disagreement statement;
- the previous arm preceded by a five-minute failure-artifact tutorial, in which participants watch one missing value flip a prediction.
A final option lets participants request the missing value, for a cost, before deciding. We recruit medical students and residents as the target population, and a larger general-adult sample for statistical power. Self-reported statistical literacy is measured because the reliance paper found it shifted switching behavior.
Baselines
The naive reference is the no-uncertainty arm. This reproduces the regime where Designing for Appropriate Reliance found 44–61% over-reliance and team accuracy (67%) below the AI alone (80%). We expect this arm to show the same pattern on tabular data, and the AI-alone accuracy on our case set anchors every comparison.
The strongest existing human-facing competitor is calibrated frequency on the pooled probability. It is the only format with a reported behavioral effect ($\chi^2 = 26.05$), and if imputation uncertainty is already folded into $\bar p$, it should capture most of the benefit. All arms see identical records, an identical model and identical calibration. The disagreement arm beats this baseline only if its over-reliance rate on wrong AI predictions for heavily imputed records is lower by a preregistered margin, and its under-reliance on correct predictions does not rise.
The strongest automated competitor removes the human from the missingness question. Uncertainty-aware models such as GRU-U report a PhysioNet 2012 AUC of 0.832, against 0.824 for BRITS, and 0.794 when the uncertainty gate is ablated, per Uncertainty-Gated Stochastic Sequential Model. A related option is deferral: the system abstains on high-$d$ records. We implement disagreement-based deferral with the same imputations and case set. We compare the accuracy of deferral combined with human decisions on the deferred records against the full disagreement-display arm, at equal coverage. For a single-snapshot tabular benchmark our model is simpler, so these AUC figures are context, not a like-for-like target.
The imputed-features-list arm is the ablation for our key ingredient. It tells users that values were imputed but not how much the fill-ins disagree. If it matches the disagreement arm, the benefit comes from provenance, not from multiple imputation.
Why not just…?
Why not just show calibrated confidence, since pooling over imputations already includes the missingness uncertainty? This is the main threat, and the offline gate exists to settle it. Pooled calibration is correct on average, but it can assign similar probabilities to records whose errors come from very different sources. If $d$ adds no error-predictive information within bins of $\bar p$, the disagreement display is redundant, and we drop the human study rather than argue around the result.
Why not just flag which values were imputed? A provenance badge is cheaper and may be all people can use. But it treats an imputed value that does not matter the same as one that flips the prediction. The imputed-features-list arm tests this directly. A tie would be a publishable negative result that recommends the simpler interface.
Why not let the system defer automatically on heavily imputed records? Automated deferral is a strong option, and it may win on accuracy alone. The human arm has two potential advantages: people can bring context the model lacks, and they can order the missing test. The deferral comparison at equal coverage, together with how often participants request missing values on high-$d$ records, will show whether those advantages are real.
Why not use native missing-value handling, such as splits on missingness in boosted trees, or direct parameter estimation, and skip imputation entirely? These methods produce one prediction per record and no spread, so there is nothing to show about how the missing values could have gone. Feature Importance Metrics in the Presence of Missing Data also cautions that splitting on missingness conflates measurement policy with physiology. We report native-missing XGBoost accuracy alongside the MICE pipeline, so readers can see that the display does not come at the cost of a weaker model. For reference, XGBoost with multiple imputation reached an AUROC of 0.78 on ICU fluid-overload prediction in Machine learning vs. traditional regression analysis. There is no reported result on Lacuna for our PhysioNet summary-feature configuration.
Decisive Checks
The first result that matters is the offline gate: does disagreement predict AI error after conditioning on calibrated probability, on PhysioNet 2012 and on the masked Adult and Default tables? The second is the interaction in the human study. Relative to the pooled calibrated-frequency arm, over-reliance should fall more on heavily imputed records than on lightly imputed ones, with no rise in under-reliance and team accuracy approaching or exceeding the AI alone. The third is whether the tutorial arm widens this effect for participants with low statistical literacy. That result would turn a single interface finding into a concrete piece of clinical AI literacy teaching.
References
- Designing for Appropriate Reliance: The Roles of AI Uncertainty Presentation, Initial User Decision, and User Demographics in AI-Assisted Decision-Making
- Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction
- Feature Importance Metrics in the Presence of Missing Data
- Correlation Visualization Under Missing Values: A Comparison Between Imputation and Direct Parameter Estimation Methods
- What do medical students actually need to know about artificial intelligence?
- Making Sense of Machine Learning: Integrating Youth’s Conceptual, Creative, and Critical Understandings of AI
- RefiDiff: Progressive Refinement Diffusion for Efficient Missing Data Imputation
- Machine learning vs. traditional regression analysis for fluid overload prediction in the ICU