Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label

Jeffrey Humpherys, Makoto JonesarXiv 2026 · arXiv

TL;DR

Medical AI models often report only the most likely diagnosis, a practice called "hardening" that discards critical uncertainty. This paper demonstrates that a model can be highly accurate at picking the top label while dangerously underestimating a patient's actual risk of death or disease.

What Problem Does This Paper Address?

In the world of machine learning, we often celebrate high accuracy. If a model correctly identifies the most likely diagnosis 90% of the time, we call it a success. However, in medicine, the "most likely" outcome isn't the only one that matters.

The authors tackle the problem of prediction hardening: when an AI takes a full probability distribution (e.g., 52% chance of reflux, 18% chance of heart attack, etc.) and exports only the top label ("Reflux"). This creates a dangerous gap between a predicted label and an observed state. Downstream systems—like triage rules or insurance authorization algorithms—then treat that label as a "fact," ignoring the "tail risks" that often carry the highest clinical consequences.

Methods

The authors use Jeffrey’s rule of conditioning to show how uncertainty should be carried forward. Instead of just looking up the risk associated with the most likely diagnosis (the "modal-state lookup"), they argue for propagated risk: a probability-weighted average of risks across all possible states.

The paper mathematically separates the failure of hardening into two distinct errors:

  1. Information Loss: Discarding the probability assigned to other, potentially more dangerous, states.
  2. Plug-in Error: Attaching the risk of patients who actually have a diagnosis to a patient who only probably has it.

They also introduce the concept of rigidity—the assumption that the risk within a specific state (like "reflux") doesn't change just because we have new information about the patient. If this assumption fails, even a perfectly calibrated model can produce incorrect risk estimates.

Datasets

Not discussed in the paper. The authors use hypothetical clinical examples and mathematical proofs rather than training on a specific real-world dataset. This is a theoretical and position paper intended to establish guardrails for the field.

Key Insights and Ideas

  • A predicted label is not an observed state. Treating an AI's "best guess" as a ground-truth fact is a category error that leads to quantifiable clinical mistakes.
  • Calibration is not a silver bullet. A model can be perfectly calibrated (meaning its 70% confidence actually corresponds to a 70% frequency) and still fail to provide the correct risk for an individual patient if the underlying "risk table" is misapplied.
  • The "Agreement Condition": The authors provide a mathematical bound (Corollary 3.2) to determine when a hard label and a full distribution will lead to the same decision. If the decision threshold is close to the predicted risk, hardening is almost guaranteed to cause errors.
  • Insurance Moral Hazard: When insurers use automated rules based on hard diagnosis codes, they may deny care to high-risk patients simply because the "most likely" code is benign, even if the aggregate risk justifies the procedure.

Key Results

The paper illustrates its points through two powerful hypothetical examples:

1. The Chest Pain Differential An AI evaluates a patient with chest pain. It finds "Gastroesophageal disease" (GD) is the most likely cause at 52%. However, it also assigns a combined 30% probability to life-threatening conditions like heart attack or aortic dissection.

Hypothetical model output for chest pain
  • Hardened Result: The system reports "Reflux," which has a 0.1% risk of death.
  • Propagated Result: The actual risk to the patient is 5.57%—more than 50 times higher. A patient who should be in the ER might be sent home based on the "most likely" label.

2. BI-RADS Imaging In breast cancer screening, Category 3 is "probably benign" (<2% risk).

Hypothetical BI-RADS risk interpretation

If a model is 70% sure a lesion is Category 3 and 30% sure it is Category 4A (6% risk), the propagated risk is 2.5%. Even though the "most likely" label is Category 3, the patient's actual risk exceeds the 2% ceiling that defines that category.

Limitations and Future Work

  • Hypothetical Nature: The results are based on illustrative numerical values. Real-world clinical event rates and model performance will vary.
  • Trustworthy Probabilities: The entire framework relies on the AI producing reliable, well-calibrated probability distributions. If the model's internal probabilities are junk, the propagated risk will be too.
  • Human Factors: The authors note that it is still an open question whether clinicians actually make better decisions when presented with complex probabilistic outputs versus simple labels.
  • Future Work: The authors propose six guardrails for developers and regulators, including requirements to state exactly what model numbers mean and to audit how much "risk" is lost during data compression.

Summary updated

Abstract

Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only the benign leading diagnosis greatly understates the patient's risk of death or cardiac arrest, so a patient who warrants evaluation could be discharged or denied imaging. In a hypothetical BI-RADS example, the most likely category is "probably benign," yet the propagated malignancy risk exceeds the ceiling that defines that category. Conclusions. A predicted label is not an observed state, and a calibrated classifier is not automatically a valid outcome model. Clinical and insurance coverage decisions should rest on the patient's outcome risk, judged against a threshold justified by the consequences of acting and not acting. Developers should state what each model output means and check that every step between the model and the decision preserves the patient's risk.
BibTeX
@misc{humpherys2026accuracy,
  author = {Jeffrey Humpherys and Makoto Jones},
  title = {{Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label}},
  year = {2026},
  eprint = {2611.01312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01312},
}
Figure