Problems in the deployment of machine-learned models in health care

Joseph Cohen, Tianshi Cao, Joseph D. Viviano, Chin‐Wei Huang, Michael Fralick, Marzyeh Ghassemi, Muhammad Mamdani, Russell Greiner, Yoshua BengioCanadian 2021 · DOI · OpenAlex

Abstract

Machine learning (ML) models often exhibit high performance in controlled research environments but fail significantly upon clinical deployment. This analysis identifies two primary causes for this "brittleness": out-of-distribution (OOD) generalization and incorrect feature attribution. By categorizing failure modes and reviewing high-profile deployment challenges, we provide a framework for understanding and mitigating the risks associated with AI in medicine, emphasizing the need for robust validation and transparent reporting.

The Gap Between Lab and Clinic

ML models frequently achieve near-perfect accuracy on specific research datasets but perform poorly when faced with real-world clinical data. Unlike human physicians, who can generalize theoretical knowledge to novel cases, ML models are strictly bound by their training distributions. When the "performance distribution" of a live hospital environment shifts away from the "training distribution," models fail in ways that are often invisible to the user, potentially leading to significant medical errors.

A Framework for Deployment Failure

Deployment problems can be broadly categorized into two themes: Out-of-Distribution (OOD) generalization and Incorrect Feature Attribution.

Out-of-Distribution (OOD) Generalization

OOD data occurs when the model encounters inputs that differ significantly from its training set. These can be classified into three distinct categories:

  1. Unrelated to the task: Non-medical images (e.g., a cat) or incorrect anatomy (e.g., a knee radiograph for a chest X-ray model).
  2. Incorrectly prepared data: Technical issues such as blurry images, rotated views, or underexposed radiographs.
  3. Selection bias: Data representing demographics or disease states not included in the original training set.
Figure 1: Taxonomy of data distribution shifts and selection biases in medical imaging.

The "Path of Least Resistance"

ML models are inherently "lazy," often identifying the most efficient statistical path to minimize loss. This leads to incorrect feature attribution, where a model relies on spurious correlations—features that are predictive but medically irrelevant. For example, a model might learn to identify pneumonia by detecting a specific hospital's watermark or the type of X-ray machine used, rather than the actual pathology.

Figure 2: An example of an unseen artifact (red arrow pointing to a pacemaker) that can lead to incorrect feature attribution if the model learns to associate the device with a specific clinical outcome.

Lessons from High-Profile Failures

The history of medical AI is punctuated by cautionary tales that illustrate these technical gaps:

  • IBM Watson for Oncology: Despite a $62 million investment, the system was suspended after providing clinical recommendations that clinicians found unhelpful or problematic.
  • Google’s Diabetic Retinopathy Initiative: In Thailand, the model struggled with real-world clinical images that were of lower quality than the high-resolution training data, causing significant workflow disruptions and frustration for staff.

Mitigation and Future Directions

To address these challenges, researchers are developing tools like autoencoders to detect OOD samples by measuring reconstruction error. If a model cannot accurately reconstruct an image, it indicates the data is unfamiliar and the model should decline to make a prediction. Furthermore, the use of model cards—standardized documentation detailing a model's training demographics and limitations—is essential for transparency before a model reaches the clinic.

Ultimately, overcoming "bad habits" in ML requires expert-led re-labeling of datasets based on true patient outcomes rather than historical clinical actions, ensuring that models learn to predict health, not just replicate past behavior.

Summary updated

Abstract

[See related articles at www.cmaj.ca/lookup/doi/10.1503/cmaj.202434][1] and [www.cmaj.ca/lookup/doi/10.1503/cmaj.210036][2] KEY POINTS In a companion article, Verma and colleagues discuss how machine-learned solutions can be developed and implemented to support medical decision-making.[1][3] Both
BibTeX
@inproceedings{cohen2021problems,
  author = {Joseph Cohen and Tianshi Cao and Joseph D. Viviano and Chin‐Wei Huang and Michael Fralick and Marzyeh Ghassemi and Muhammad Mamdani and Russell Greiner and Yoshua Bengio and Joseph Paul Cohen},
  title = {{Problems in the deployment of machine-learned models in health care}},
  booktitle = {Canadian},
  year = {2021},
  doi = {10.1503/cmaj.202066},
}
Figure