Introduction
In clinical research, "labeling" patient outcomes—such as determining if cancer has recurred or if a patient has developed specific side effects—remains a significant bottleneck. Traditionally, this process requires highly trained experts to manually review thousands of pages of clinical notes, pathology reports, and imaging. This manual review is not only slow and expensive but also prone to human error, even in "gold standard" research registries.
RadOnc-GPT addresses this challenge by introducing an autonomous, GPT-4o-based agent designed to automate the labeling of patient outcomes in radiation oncology. By directly querying electronic health records (EHR) and clinical notes, the system achieves near-expert accuracy and serves as a real-time auditor, identifying discrepancies in existing research data.
System Architecture and Methods
The core innovation of RadOnc-GPT lies in its transition from standard Retrieval-Augmented Generation (RAG) to an autonomous agentic system. While RAG often loses context by breaking documents into arbitrary "chunks," RadOnc-GPT utilizes a function-calling architecture.
The system is equipped with a suite of "whitelisted" functions that allow it to autonomously decide which data to pull based on the specific clinical task. For instance, it can call get_patient_clinical_notes or get_pathology_reports to gather the necessary evidence for a diagnosis.
Key technical features include:
- Function-Calling Architecture: Instead of standard RAG, the agent uses targeted functions to maintain temporal bounds and clinical context.
- LLM Task Streaming: A wrapper program that enables parallel processing of patient cohorts, handling data retrieval logic and generating structured JSON outputs.
- Evidence-Based Prompting: For complex tasks like Mandibular Osteoradionecrosis (ORN), the agent is prompted to follow established medical criteria (e.g., the Marx staging system) and provide a balanced argument before reaching a conclusion.
Datasets and Evaluation
The study evaluated RadOnc-GPT using data from 895 unique patients at the Mayo Clinic. The evaluation was split into two tiers:
- Tier 1 (QA): 500 randomly selected radiation therapy patients (2015–2025) to establish foundational trust in data retrieval.
- Tier 2 (Complex Outcomes): Focused on specific clinical endpoints, including an ORN cohort of 233 Head-and-Neck Cancer (HNC) patients, a Prostate Recurrence cohort (80 patients), and an HNC Recurrence cohort (82 patients).
Initial labels were sourced from existing research registries. Crucially, all discrepancies between the model's output and the registry were adjudicated by independent expert radiation oncologists to determine the true "ground truth."
Key Insights: The LLM as an Auditor
A significant finding of the study was the "Auditor Effect." RadOnc-GPT did not just replicate existing labels; it actively identified errors in the human-curated registries. Out of 48 initial discrepancies, 63% were found to be errors in the original "gold standard" registry. This suggests that LLMs can significantly improve the quality of research data by acting as a secondary, tireless reviewer.
Furthermore, the study demonstrated that targeted function calling is superior to vector-based RAG in structured EHR environments. The agent also showed remarkable generalization, using the same prompt to detect cancer recurrence across different disease sites (prostate and head-and-neck) without site-specific fine-tuning.
Results and Performance
RadOnc-GPT demonstrated high accuracy across all tested domains:
- Structured Data: Achieved 100% accuracy in retrieving demographic fields and 99.4% accuracy in radiation treatment counts.
- Complex Outcomes (Post-Adjudication):
- ORN Detection: 95.2% accuracy with 100% recall.
- Prostate Cancer Recurrence: 95.0% accuracy.
- HNC Recurrence: 96.3% accuracy.
In terms of efficiency, the system processed individual patients in 10–30 seconds. This represents a massive speedup compared to manual chart reviews, which typically take 20–60 minutes per patient.
Limitations and Future Directions
While promising, the study was conducted within a single-center ecosystem (Mayo Clinic), and performance may vary in institutions with different EHR structures. Additionally, the inference costs of high-end models like GPT-4o remain a consideration for large-scale hospital deployments. Future work will focus on benchmarking against open-source models and optimizing prompts to enable smaller, more cost-efficient models to achieve comparable performance.
Summary updated