Making the invisible visible: what can we do about biased AI in medical devices?

Margaret Whitehead, Enitan D. Carrol, Frank Kee, Chris HolmesBMJ 2023 · DOI · OpenAlex

This summary provides an overview of the opinion piece published in The BMJ regarding the risks and mitigation strategies for algorithmic bias in medical technology.

TL;DR

AI in medical devices often operates as an "invisible" layer of decision-making that can entrench racial, gender, and socioeconomic biases. To prevent systemic harm, the authors argue for radical transparency in training data, the elimination of flawed biological proxies, and robust regulatory oversight.

What Problem Does This Paper Address?

As AI becomes embedded in everything from cancer screening to surgical genomics, its decision-making processes remain largely opaque to both clinicians and patients. This "invisibility" masks fundamental biases that disproportionately harm marginalized groups.

The authors identify two primary drivers of this problem:

  1. Invisible People: Certain demographics (women, ethnic minorities, low-income populations) are under-represented in the datasets used to train and validate AI.
  2. Invisible Inputs: The logic or "proxies" used by algorithms—especially proprietary commercial ones—are often undisclosed, leading to systemic discrimination based on flawed assumptions.

Methods

This is an opinion and perspective piece rather than a primary research study. The authors synthesize evidence from recent clinical failures and algorithmic audits to propose a multi-stakeholder framework for "making the invisible visible." Their approach focuses on:

  • Case Study Analysis: Examining real-world failures in dermatology, cardiology, and nephrology.
  • Policy Recommendations: Proposing specific actions for developers, dataset curators, and regulators to ensure health equity.

Datasets

The paper does not introduce a new dataset; instead, it critiques the current state of data used in medical AI:

  • Skin Cancer Imaging: The authors note that common AI-assisted imaging devices are "trained" predominantly on images of lighter skin tones, leading to potential under-diagnosis in patients with darker skin.
  • Historical Clinical Data: They highlight how using historical data for heart disease training can bake in existing biases, such as the fact that men are 2.5 times more likely to be referred to a cardiologist than women presenting with the same symptoms.
  • Socioeconomic Proxies: They critique the use of healthcare spending as a proxy for health need, which inherently biases against lower-income patients who have less access to care.

Key Insights and Ideas

  • The Proxy Trap: A major "aha moment" involves a US commercial risk tool that used past healthcare costs as a measure of ill health. Because affluent people spend more on healthcare regardless of their actual health status, the algorithm erroneously prioritized them for intensive treatment over poorer, sicker patients.
  • Biological Myths in Code: The paper highlights "race correction factors" (e.g., in serum creatine levels for kidney disease) that are built into diagnostic tools. These are based on outdated, flawed beliefs about biological differences between ethnic groups and can lead to significant delays in treatment for Black patients.
  • The Invisibility of Commercial AI: Proprietary "black box" models are particularly dangerous because their inputs are often hidden for commercial reasons, preventing independent audits for bias or flawed assumptions.
  • Generative AI Risks: The authors warn that the "looming revolution" of Large Language Models (LLMs) in healthcare will exacerbate these issues unless equity is prioritized in their development.

Key Results

While the paper is qualitative, it cites several striking quantitative findings from the literature:

  • Gender Bias: Men are 2.5 times more likely to be referred for specialist cardiac management than women, despite equal proportions presenting with chest pain.
  • Algorithmic Failure: In the Netherlands, a biased AI system for childcare benefits led to the false accusation of tens of thousands of parents (mostly from low-income or ethnic minority backgrounds) of fraud.
  • Clinical Impact: Race-based "correction" in kidney function algorithms has historically delayed treatment for Black patients until their disease was significantly more advanced.

Limitations and Future Work

The authors acknowledge that achieving transparency is difficult when it conflicts with commercial interests and proprietary secrets. They suggest several paths forward:

  • Public Involvement: Patients and the public must be involved at every stage of device development to help frame and prioritize equity issues.
  • Standardization: There is a need for "dataset standards" (referencing the Standing Together initiative) to support diversity and generalizability.
  • Regulatory Accountability: Regulators must require transparent descriptions of datasets during the approval process and perform continual monitoring of devices once they are in use.
  • Education: Improving the "health equity literacy" of AI developers and the "AI literacy" of health professionals is essential for long-term systemic change.

Summary updated

Abstract

AI medical applications and potential harm to health, write
BibTeX
@inproceedings{whitehead2023making,
  author = {Margaret Whitehead and Enitan D. Carrol and Frank Kee and Chris Holmes},
  title = {{Making the invisible visible: what can we do about biased AI in medical devices?}},
  booktitle = {BMJ},
  year = {2023},
  doi = {10.1136/bmj.p1893},
}

Loading related content...

Figure