Regulating radiology AI medical devices that evolve in their lifecycle

Camila González, Moritz Fuchs, Daniel Pinto dos Santos, Philipp Matthies, Manuel Trenz, Maximilian Grüning, Akshay Chaudhari, David B. Larson, Ahmed E. Othman, Moon Kim, Felix Nensa, Anirban MukhopadhyayarXiv 2024 · arXiv · OpenReview

TL;DR

The paper argues that traditional, "locked" AI medical devices are becoming obsolete due to evolving data distributions. It highlights new regulatory frameworks in the EU and US that enable "lifelong learning" AI systems, allowing for safe, post-market updates without full re-approval, provided robust monitoring and transparent processes are in place.

What Problem Does This Paper Address?

Medical AI, particularly in radiology, faces a critical challenge: the "distribution shift." Over time, the data used to train these AI models can drift from the real-world data they encounter in clinical practice. This drift can be caused by changes in patient demographics, new imaging equipment, or evolving disease patterns. Unlike human radiologists who can adapt their expertise, AI models are brittle and their performance degrades significantly when faced with data that differs from their training set. This leads to outdated, less effective AI tools remaining in use for too long because the regulatory process for updating them (requiring full re-approval) was prohibitively slow and expensive.

Methods

The paper doesn't introduce new algorithms or models. Instead, it focuses on the regulatory and operational framework required for AI medical devices to evolve safely over their lifecycle. It analyzes recent regulatory developments, specifically the EU's Artificial Intelligence Act and the FDA's Predetermined Change Control Plan (PCCP), which aim to facilitate continuous learning and updates. The authors outline the "building blocks" necessary for dynamic AI systems, emphasizing:

  • Structured Reporting: Standardized data formats to ensure consistency and facilitate model retraining.
  • Continual Learning Techniques: While not detailing specific algorithms, the paper acknowledges the need for methods that can adapt to new data without catastrophic forgetting (losing previously learned knowledge).
  • Uncertainty Quantification and OOD Detection: Mechanisms for the AI to signal when it's encountering data outside its reliable operating range.
  • Transparency and Traceability: Clear documentation of changes, data, and model behavior.
  • Real-world Performance Monitoring: Continuous tracking of AI performance in clinical settings.
  • Active Radiologist Involvement: Ensuring human oversight and feedback loops.

Datasets

Not discussed in the paper. The paper focuses on the regulatory and operational aspects of AI lifecycle management rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • The "Locked AI" Paradigm is Unsustainable: AI models that cannot be updated post-deployment are destined to become less effective over time due to inevitable data drift.
  • Regulatory Evolution is Key: The recent shift towards "lifecycle regulatory protocols" (like the FDA's PCCP and the EU's AI Act) is a critical enabler for dynamic AI medical devices.
  • Trust is Built on Transparency and Monitoring: For AI to be safely updated, manufacturers must provide clear documentation of their data handling, retraining processes, and implement robust real-world quality monitoring.
  • Continual Learning Requires More Than Just Data: Adapting AI models to new data without degrading performance or introducing new biases is a complex technical challenge that requires careful consideration of techniques like rehearsal-based methods or ensemble approaches.
  • Radiologists Remain Essential: The future of AI in radiology is collaborative, with radiologists acting as crucial partners in quality assurance and data annotation.

Key Results

The paper doesn't present quantitative results from experiments. Its primary contribution is a qualitative analysis and synthesis of regulatory trends and operational requirements. The "key results" are the identification and articulation of the necessary components for a successful "lifelong learning" AI medical device framework, as mandated by emerging regulations.

Limitations and Future Work

The authors acknowledge that while the new regulatory frameworks provide a foundation, critical gaps remain. These include:

  • Lack of Standardized Metrics: The definition of appropriate evaluation metrics, datasets, and performance deterioration thresholds is not clearly defined and is left to manufacturers.
  • Handling Performance Trade-offs: How to manage situations where performance improves for some sub-populations but degrades for others is unclear.
  • Comparison of Algorithms: The regulation lacks clear definitions for diagnostic tasks and mechanisms for comparing similar algorithms.
  • Robust Safety and Performance Characterization: Further work is needed on how to robustly characterize the safety and performance of these evolving systems.

Future work will likely involve the industry and regulatory bodies establishing more objective standards for these areas as the field matures.

Summary updated

Abstract

Over time, the distribution of medical image data drifts due to factors such as shifts in patient demographics, acquisition devices, and disease manifestations. While human radiologists can adjust their expertise to accommodate such variations, deep learning models cannot. In fact, such models are highly susceptible to even slight variations in image characteristics. Consequently, manufacturers must conduct regular updates to ensure that they remain safe and effective. Performing such updates in the United States and European Union required, until recently, obtaining re-approval. Given the time and financial burdens associated with these processes, updates were infrequent, and obsolete systems remained in operation for too long. During 2024, several regulatory developments promised to streamline the safe rollout of model updates: The European Artificial Intelligence Act came into effect last August, and the Food and Drug Administration (FDA) issued final marketing submission recommendations for a Predetermined Change Control Plan (PCCP) in December. We provide an overview of these developments and outline the key building blocks necessary for successfully deploying dynamic systems. At the heart of these regulations - and as prerequisites for manufacturers to conduct model updates without re-approval - are clear descriptions of data collection and re-training processes, coupled with robust real-world quality monitoring mechanisms.
BibTeX
@misc{gonzalez2024regulating,
  author = {Camila González and Moritz Fuchs and Daniel Pinto dos Santos and Philipp Matthies and Manuel Trenz and Maximilian Grüning and Akshay Chaudhari and David B. Larson and Ahmed E. Othman and Moon Kim and Felix Nensa and Anirban Mukhopadhyay},
  title = {{Regulating radiology AI medical devices that evolve in their lifecycle}},
  year = {2024},
  eprint = {2412.20498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20498},
}
Figure