Abstract
The ability of pre-trained large language models (LLMs) to rapidly master novel natural language processing tasks holds transformative potential. However, pre-trained LLMs often struggle to achieve high performance in specialized domains such as oncology and have the tendency to deliver incorrect information confidently (“hallucinate”), limiting their utility in such contexts. Retrieval-augmented generation (RAG) addresses this limitation by dynamically incorporating authoritative, domain-specific knowledge directly into the LLM’s inference process. In this study, we demonstrate the exceptional performance of a minimalist RAG pipeline on radiation oncology board-style examinations. Leveraging a curated knowledge base sourced from Gunderson & Tepper’s Clinical Radiation Oncology and NCCN guidelines, our model achieved an outstanding accuracy of 91.5% on the 2021 American College of Radiology (ACR) TXIT examination, markedly exceeding previous benchmarks. Crucially, our model exhibited robust self-awareness regarding its knowledge boundaries, with incorrect answers reliably flagged by low confidence scores.
Introduction
While general-purpose LLMs like GPT-4 demonstrate impressive capabilities, they are prone to "hallucinations"—confidently stating false information. In radiation oncology, where treatment involves complex dose calculations and strict technical protocols, such errors can be life-threatening. General models often lack the niche, up-to-date knowledge found in specialized medical literature and fail to provide a "paper trail" for their reasoning. This research sought to bridge this gap by creating a system that actively consults trusted medical sources before generating responses, ensuring both accuracy and interpretability.
Methodology
Data Processing and Knowledge Base Construction
The foundation of the system is a specialized "open book" knowledge base. The data collection phase involved extracting text from gold-standard textbooks, the latest NCCN Guidelines, and a supplementary set of 1,000 PubMed articles. This text was preprocessed into ~512 token chunks and vectorized using OpenAI’s text-embedding-3-large model. The resulting embeddings were stored in a ChromaDB vector database to enable semantic search.
Multi-Agent RAG Pipeline
The researchers implemented a multi-agent RAG pipeline designed for high-stakes clinical reasoning. The workflow consists of four primary stages:
- Query Reformulation: Extracting key entities (disease, treatment, etc.) from the user's question.
- Multi-Agent Search: Five independent agents generate semantically rich queries to search the curated database.
- Evidence Retrieval & Inference: The system retrieves the top-5 most relevant text chunks. Each agent then independently infers an answer based strictly on that retrieved text, providing a natural-language explanation and a confidence score (1-10).
- Majority Voting: The final answer is determined by a majority vote among the agents to mitigate individual model biases.
Results and Discussion
State-of-the-Art Performance
The RAG-enhanced model achieved a record-breaking 91.5% accuracy on the 2021 ACR TXIT examination, a significant improvement over the previous benchmark of 74%. A key finding was the "leapfrogging" effect: older or "lighter" models equipped with RAG (e.g., GPT-4-turbo with RAG at 79.38%) outperformed newer, larger models running without it (e.g., GPT-4o at 77.3%). This suggests that RAG is a powerful force multiplier for specialized tasks.
Actionable Self-Awareness
One of the most significant advancements was the model's ability to "know what it doesn't know." There was a high correlation between confidence scores and actual accuracy. For the top-performing model, the mean confidence for correct answers was 7.36/10, while incorrect answers averaged just 4.12/10. With an AUC of 0.91 for discriminating between correct and incorrect responses, the system can effectively flag low-confidence answers for human review, enhancing clinical safety.
The Impact of Data Quality
Interestingly, adding 1,000 uncurated PubMed articles to the knowledge base actually decreased performance for advanced models. This indicates that "noise" from broad literature can sometimes dilute the high-quality "signal" provided by consensus-based textbooks and guidelines, highlighting the importance of curated knowledge bases in medical AI.
Conclusion
Integrating domain-specific knowledge via RAG significantly enhances LLM performance in radiation oncology, enabling reliable confidence scoring and superior accuracy. While current limitations include a text-only constraint and the use of multiple-choice benchmarks, future work will focus on multimodal capabilities (incorporating CT/MRI data) and testing against real-world clinical scenarios. This scalable approach holds great promise for clinical decision support and medical education.
Summary updated