Regulatory considerations for medical imaging AI/ML devices in the United States: concepts and challenges

Nicholas Petrick, Weijie Chen, Jana G. Delfino, Brandon D. Gallas, Yanna Kang, Daniel M. Krainak, Berkman Sahiner, Ravi K. SamalaJournal of Medical Imaging 2023 · DOI

TL;DR

This paper provides a comprehensive overview of the U.S. Food and Drug Administration's (FDA) regulatory framework for artificial intelligence and machine learning (AI/ML) devices in medical imaging. It details the processes, data considerations, and challenges involved in bringing these innovative technologies to market, emphasizing the FDA's commitment to ensuring patient safety while fostering innovation.

What Problem Does This Paper Address?

The rapid advancement of AI/ML in medical imaging presents a unique set of challenges for regulatory bodies. Developers need a clear understanding of how to navigate the FDA's approval processes, which are designed for traditional medical devices but must now accommodate the complexities of adaptive algorithms, vast datasets, and evolving performance characteristics inherent in AI/ML. This paper addresses the need for a consolidated resource that introduces AI/ML developers to the FDA's regulatory concepts, processes, and fundamental assessments, while also highlighting ongoing challenges and the agency's efforts to adapt.

Methods

The paper outlines the FDA's regulatory framework, which classifies devices based on risk (Classes I, II, III) and determines the appropriate regulatory pathway (Premarket Notification [510(k)], Premarket Approval [PMA], or De Novo). It emphasizes the importance of a detailed "model description" that includes input data, engineered features, pre/post-processing steps, network architecture, and development processes. Crucially, it details the critical role of data in AI/ML submissions, including considerations for data collection, representativeness, and independence between training and testing sets. The paper also discusses nonclinical testing, focusing on standalone performance evaluation and the necessity of metrics like AUROC, and introduces Multi-Reader Multi-Case (MRMC) studies as a key method for evaluating AI/ML devices in a clinical context.

Datasets

Not explicitly detailed in the paper, as the focus is on the regulatory process rather than specific model training. However, the paper emphasizes the critical need for large, representative datasets for AI/ML training and testing, drawing from various sources including manufacturer-collected data, private collections, public datasets, and potentially synthetic data.

Key Insights and Ideas

  • Risk-Based Classification is Key: The FDA's regulatory approach hinges on the risk associated with an AI/ML device, determined by its intended use and technological characteristics.
  • Data is Paramount: The quality, representativeness, and independence of training and testing data are fundamental to developing robust and generalizable AI/ML models.
  • Independence is Non-Negotiable: Ensuring that test data is independent of training data (ideally across different sites and times) is crucial to avoid optimistically biased performance estimates.
  • MRMC Studies Bridge the Gap: Multi-Reader Multi-Case studies are essential for evaluating how AI/ML devices perform in the hands of clinicians, mimicking real-world use.
  • FDA is Actively Adapting: The FDA is not static; it's actively developing policies, conducting research, and engaging with stakeholders to refine its regulatory science and processes for AI/ML.

Key Results

The paper doesn't present quantitative results from specific AI/ML device evaluations. Instead, it highlights the FDA's ongoing efforts and the types of information required for submissions. It notes that over five hundred medical devices incorporating AI/ML technology have received marketing authorization, with the majority analyzing radiological image data. The paper also points to the FDA's development of guidance documents and research initiatives aimed at addressing AI/ML-specific challenges.

Limitations and Future Work

The paper acknowledges the inherent challenges in regulating AI/ML, particularly concerning:

  • Algorithm Robustness: Ensuring AI/ML models are reliable and perform consistently across diverse patient populations, clinical settings, and imaging devices.
  • Data Governance: Establishing effective policies for data quality, security, and the use of reference standards.
  • Transparency and Explainability: Determining the appropriate level of transparency needed for users and regulators to understand AI/ML outputs.
  • Continuously Learning Models: Developing regulatory frameworks for AI/ML devices that adapt and learn over time.

Future work by the FDA and the broader research community will focus on developing novel assessment methods, improving AI/ML robustness, enhancing transparency, and creating regulatory science tools to support the evaluation of these rapidly evolving technologies. The agency is actively seeking public comment on proposed frameworks for managing modifications to AI/ML devices.

Summary updated

Abstract

FDA's AI/ML regulatory and scientific efforts support the joint goals of ensuring patients have access to safe and effective AI/ML devices over the entire device lifecycle and stimulating medical AI/ML innovation.
BibTeX
@article{petrick2023regulatory,
  author = {Nicholas Petrick and Weijie Chen and Jana G. Delfino and Brandon D. Gallas and Yanna Kang and Daniel M. Krainak and Berkman Sahiner and Ravi K. Samala},
  title = {{Regulatory considerations for medical imaging AI/ML devices in the United States: concepts and challenges}},
  journal = {Journal of Medical Imaging},
  year = {2023},
  doi = {10.1117/1.jmi.10.5.051804},
}
Figure