Introduction
Large Language Models (LLMs) like GPT-4 have demonstrated significant potential in healthcare applications. However, their deployment in clinical settings is hindered by a critical flaw: hallucinations. Generating fictitious or inaccurate medical information can lead to severe consequences in patient care. To address this, researchers have turned to Retrieval-Augmented Generation (RAG). The primary challenge in grounding these models is the heterogeneity of medical knowledge, which exists in structured knowledge graphs, semi-structured databases, and unstructured clinical notes or textbooks.
The ClinicalRAG Framework
The authors propose ClinicalRAG, a multi-agent pipeline designed to bridge the gap between raw, disparate medical data and LLM reasoning. Unlike traditional RAG systems that struggle with diverse data formats, ClinicalRAG utilizes specialized agents to process and synthesize information.
The pipeline consists of four distinct stages:
- Medical Entity Extraction (MEE): Parses the patient's "chief complaint" to identify key entities such as symptoms, potential diseases, and treatments.
- Heterogeneous Knowledge Index (HKI): Performs vector-based similarity searches across three silos: Knowledge Graphs (G), Knowledge Bases (B), and Online Resources (O).
- Knowledge to Natural Language (KNL): Converts retrieved data—including structured graph triples—into natural language descriptions. This ensures the LLM receives "pre-digested" information in its most effective reasoning format.
- Knowledge Reference Generation (KRG): Deduplicates and sorts retrieved information by relevance, providing a standardized reference list to support the final diagnosis.
Experimental Setup
The framework was evaluated using the CBLUE EHR Dataset, comprising 2,000 records of patient complaints and ground-truth diagnoses. Structured data was sourced from DiseaseKG, an open-source Chinese medical knowledge graph, while medical textbooks and Wikipedia provided unstructured knowledge.
Key Results and Analysis
ClinicalRAG was tested against several mainstream LLMs, including GPT-3.5, GPT-4, and Llama-2 variants. The results demonstrate a consistent performance boost across all models compared to direct classification.
Performance Gains
As shown in the results, GPT-4's diagnostic accuracy rose from 82.78% to 84.94% when using the ClinicalRAG pipeline. The system also outperformed specialized medical models like MedKPL (77.78%) and standard BERT-based classifiers (74.07%).
The Importance of Natural Language Synthesis
A key insight from the ablation studies was the criticality of the KNL agent. Converting heterogeneous data into natural language before prompt fusion was the most significant factor in performance. Removing this component led to a 1.53% drop in accuracy, confirming that natural language serves as a "universal interface" for LLM reasoning.
Conclusion and Future Work
ClinicalRAG effectively reduces hallucinations by providing medical-knowledge-aware context. While the multi-agent approach improves accuracy and verifiability, it introduces computational overhead and higher latency. Future work will focus on optimizing these processes for real-time clinical environments and expanding the framework's validation across different languages and diverse Electronic Health Record (EHR) systems.
Summary updated