Role and Use of Race in Artificial Intelligence and Machine Learning Models Related to Health

Martin C. Were, Ang Li, Bradley Malin, Zhijun Yin, Joseph Coco, Benjamin Collins, Ellen Wright Clayton, Laurie L. Novak, Rachele Hendricks‐Sturrup, Abiodun Oluyomi, Shilo Anders, Chao YanJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper provides a comprehensive framework for understanding and addressing the complex role of race in health-related AI and ML models. It argues for a systematic, life-cycle approach to identify and mitigate potential harms, emphasizing that race is a social construct and should not be conflated with genetics.

What Problem Does This Paper Address?

The increasing integration of AI and ML in healthcare presents a critical challenge: how to ensure these technologies do not perpetuate or exacerbate existing health disparities, particularly those related to race. While race is a powerful predictor of health outcomes and experiences of discrimination, its inclusion in AI/ML models is fraught with complexity. Models can inadvertently encode biases, leading to unequal treatment or inaccurate predictions for certain racial groups. Existing efforts to address this are often fragmented, lacking a holistic framework to guide developers, policymakers, and users through the entire AI/ML lifecycle. This paper aims to fill that gap by offering a systematic approach to critically examine the role and use of race in health AI/ML.

Methods

The paper proposes an AI/ML life cycle framework, adapted for a patient-centered approach, to systematically analyze the challenges associated with race. This framework consists of six key stages: Purpose, Population, Data, Model Development, Model Validation, and Model Deployment. For each stage, the authors outline specific "points to consider" and highlight potential pitfalls and best practices. This structured approach allows for a comprehensive examination of how race is conceptualized, collected, used, and evaluated throughout the development and implementation of AI/ML models in healthcare. The methodology is primarily a conceptual analysis and synthesis of existing literature and ethical considerations, rather than empirical testing of new algorithms.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Race is a Social Construct, Not a Biological Proxy: The paper strongly emphasizes that race is a social construct with no inherent biological basis. It warns against conflating race with genetic variation or ancestry, as this can lead to flawed assumptions and biased models.
  • Holistic Life Cycle Approach is Crucial: Addressing racial bias in AI/ML requires a systematic approach that spans the entire AI/ML life cycle, from defining the model's purpose to its deployment and ongoing monitoring. Ignoring any stage can lead to unintended negative consequences.
  • Context Matters: The meaning and applicability of racial categories vary significantly across different locations, cultures, and time periods. Models must account for this variability and avoid assuming universal definitions.
  • Community Engagement is Non-Negotiable: For AI/ML models that impact health outcomes, engaging with affected racial groups is essential. This ensures that the models address the needs of target communities and that risks and benefits are equitably distributed.
  • Data Quality and Representativeness are Paramount: Inadequate or biased data is a primary source of racial bias in AI/ML. The paper stresses the importance of representative, high-quality data and careful consideration of data labeling and preprocessing.
  • Algorithmic Fairness vs. Accuracy Trade-offs: The paper acknowledges the inherent tension between achieving algorithmic fairness and maintaining model accuracy. Stakeholders must engage in context-specific discussions to determine acceptable trade-offs.
  • Beyond Race: Nonmedical Drivers of Health (NMDOH): The paper suggests exploring whether NMDOH (social, economic, environmental factors) can augment or even replace race as a variable in models, as these factors often underlie observed racial disparities.

Key Results

This paper does not present empirical results from a specific AI/ML model. Instead, its key "results" are the insights and the proposed framework itself. The primary outcome is a structured guide for stakeholders to critically evaluate and address the role of race in health AI/ML. The paper identifies numerous "points to consider" across the AI/ML life cycle, offering actionable guidance for researchers, developers, and policymakers.

Limitations and Future Work

The authors acknowledge that the proposed framework is a starting point and needs to be applied to real-world case studies to evaluate its utility. They also highlight the ongoing challenge of balancing model accuracy with fairness, and the need for further research into effective strategies for achieving both. Future work should focus on developing and testing specific methods for bias mitigation within each stage of the AI/ML life cycle, as well as exploring how to best incorporate NMDOH and genetic data to create more equitable AI/ML models. The paper also points to the nascent state of regulation for AI/ML models, underscoring the need for robust governance mechanisms.

Summary updated

Abstract

The role and use of race within health-related artificial intelligence (AI) and machine learning (ML) models have sparked increasing attention and controversy. Despite the complexity and breadth of related issues, a robust and holistic framework to guide stakeholders in their examination and resolution remains lacking. This perspective provides a broad-based, systematic, and crosscutting landscape analysis of race-related challenges, structured around the AI and ML life cycle and framed through "points to consider" to support inquiry and decision-making.
BibTeX
@article{were2025role,
  author = {Martin C. Were and Ang Li and Bradley Malin and Zhijun Yin and Joseph Coco and Benjamin Collins and Ellen Wright Clayton and Laurie L. Novak and Rachele Hendricks‐Sturrup and Abiodun Oluyomi and Shilo Anders and Chao Yan},
  title = {{Role and Use of Race in Artificial Intelligence and Machine Learning Models Related to Health}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/73996},
}
Figure