IR-GPT: AI Foundation Models to Optimize Interventional Radiology

Jacqueline Brenner, James T Anibal, Lindsey Hazen, Gyung Hyuck Ko, Hannah Huth, Daguang Xu, Sheng Xu, Bradford J. WoodCardioVascular 2025 · DOI

Abstract

Foundation artificial intelligence (AI) models are capable of complex tasks that involve text, medical images, and many other types of data, but have not yet been customized for procedural medicine. This report reviews prior work in deep learning related to interventional radiology (IR), identifying barriers to generalization and deployment at scale. Moreover, this report outlines the potential design of an "IR-GPT" foundation model to provide a unified platform for AI in IR, including data collection, annotation, and training methods—while also contextualizing challenges and highlighting potential downstream applications.

Introduction: The Need for a Unified Model in IR

Interventional Radiology (IR) represents the "special operations" of modern medicine, characterized by image-guided, minimally invasive procedures where clinicians must make rapid, high-stakes decisions based on real-time fluoroscopy, patient vitals, and pre-procedural imaging. Despite the rapid advancement of AI in diagnostic radiology, its application in the IR suite remains fragmented.

Current AI solutions in IR are largely "piecemeal," focusing on narrow, task-specific goals such as segmenting a specific organ or detecting a single type of lesion. These models lack the domain knowledge and contextual awareness required to understand the holistic flow of a surgical procedure. For a single complex intervention, a hospital might theoretically need dozens of specialized models, a requirement that is computationally prohibitive and practically unmanageable for clinicians mid-procedure. To bridge this gap, we propose IR-GPT, a multimodal foundation model designed to serve as a generalist assistant for the IR environment.

Conceptual workflow of a general GPT-based model for medical data processing and output detection.

The IR-GPT Framework

The proposed IR-GPT architecture moves away from the "one-task, one-organ" paradigm toward a unified system capable of processing diverse, multimodal data streams. The framework is built upon a three-stage training pipeline designed to instill both general medical knowledge and specific procedural expertise:

  1. Multimodal Pre-training: The model begins with large-scale weights derived from general datasets. This establishes a baseline understanding of language, visual patterns, and basic medical concepts.
  2. Supervised Learning via Procedural Narrations: This novel stage addresses the "annotation bottleneck." By recording audio narrations of surgical teams describing their actions in real-time, an LLM can convert these descriptions into structured Q&A pairs. This allows the model to learn the relationship between specific visual cues (e.g., a 3D fluoroscopy frame) and clinical decision-making (e.g., optimal catheter positioning).
  3. Reinforcement Learning with Human Feedback (RLHF): To ensure clinical safety, expert interventional radiologists rank the model's outputs. This process refines the model’s ability to provide empathetic, technically accurate, and contextually appropriate recommendations.
The proposed IR-GPT architecture, illustrating the integration of multimodal inputs into a unified LLM for procedural decision support.

Key Insights and Data Strategy

A primary challenge in developing IR-GPT is the lack of a centralized "Big Data" repository for interventional procedures, unlike the well-established benchmarks in diagnostic radiology such as MIMIC-CXR. To overcome this, we advocate for a coordinated consortium to collect a multimodal dataset comprising image, text, and audio data from live IR procedures.

From Snapshots to Temporal Sequences

Unlike diagnostic radiology, which typically interprets static images, IR is inherently iterative. IR-GPT’s primary strength lies in its ability to maintain temporal context. By understanding the sequence of events leading up to a specific moment, the model can provide predictive guidance for the next steps of a procedure. Furthermore, the model's transformer-based architecture allows it to remain invariant to the size or order of inputs, making it uniquely suited for the messy, variable data types found in the EHR and live imaging suites.

Potential Applications and Future Directions

While IR-GPT remains a conceptual vision, its potential applications span the entire clinical workflow:

  • Pre-procedural: Assisting tumor boards by synthesizing patient history and imaging.
  • Intra-procedural: Providing real-time needle pathway predictions and radiation dose optimization.
  • Post-procedural: Automating the generation of procedural notes and follow-up recommendations.

Challenges to Deployment

Several significant hurdles remain. The speed constraint is paramount; a model that requires multi-turn dialogue is ineffective in a fast-paced IR suite where seconds matter. Additionally, issues regarding data silos, algorithmic bias, and patient privacy—particularly concerning the recording of operative room audio—must be addressed through robust anonymization and ethical frameworks. Finally, unlike text-based LLMs where "hallucinations" are merely inconvenient, errors in an IR-GPT model could be fatal, necessitating rigorous, multi-point validation throughout the procedural lifecycle.

Summary updated

Abstract

Foundation artificial intelligence (AI) models are capable of complex tasks that involve text, medical images, and many other types of data, but have not yet been customized for procedural medicine. This report reviews prior work in deep learning related to interventional radiology (IR), identifying barriers to generalization and deployment at scale. Moreover, this report outlines the potential design of an "IR-GPT" foundation model to provide a unified platform for AI in IR, including data collection, annotation, and training methods-while also contextualizing challenges and highlighting potential downstream applications.
BibTeX
@inproceedings{brenner2025ir,
  author = {Jacqueline Brenner and James T Anibal and Lindsey Hazen and Gyung Hyuck Ko and Hannah Huth and Daguang Xu and Sheng Xu and Bradford J. Wood},
  title = {{IR-GPT: AI Foundation Models to Optimize Interventional Radiology}},
  booktitle = {CardioVascular},
  year = {2025},
  doi = {10.1007/s00270-024-03945-0},
}
Figure