ClinicalRAG: Enhancing Clinical Decision Support through Heterogeneous Knowledge Retrieval

Yuxing Lu, Xukai Zhao, Jinzhuo Wang2024 · DOI

Introduction

Large Language Models (LLMs) like GPT-4 have demonstrated significant potential in healthcare applications. However, their deployment in clinical settings is hindered by a critical flaw: hallucinations. Generating fictitious or inaccurate medical information can lead to severe consequences in patient care. To address this, researchers have turned to Retrieval-Augmented Generation (RAG). The primary challenge in grounding these models is the heterogeneity of medical knowledge, which exists in structured knowledge graphs, semi-structured databases, and unstructured clinical notes or textbooks.

The ClinicalRAG Framework

The authors propose ClinicalRAG, a multi-agent pipeline designed to bridge the gap between raw, disparate medical data and LLM reasoning. Unlike traditional RAG systems that struggle with diverse data formats, ClinicalRAG utilizes specialized agents to process and synthesize information.

The ClinicalRAG multi-agent pipeline architecture, detailing the flow from user input to final diagnosis through specialized agents.

The pipeline consists of four distinct stages:

  1. Medical Entity Extraction (MEE): Parses the patient's "chief complaint" to identify key entities such as symptoms, potential diseases, and treatments.
  2. Heterogeneous Knowledge Index (HKI): Performs vector-based similarity searches across three silos: Knowledge Graphs (G), Knowledge Bases (B), and Online Resources (O).
  3. Knowledge to Natural Language (KNL): Converts retrieved data—including structured graph triples—into natural language descriptions. This ensures the LLM receives "pre-digested" information in its most effective reasoning format.
  4. Knowledge Reference Generation (KRG): Deduplicates and sorts retrieved information by relevance, providing a standardized reference list to support the final diagnosis.

Experimental Setup

The framework was evaluated using the CBLUE EHR Dataset, comprising 2,000 records of patient complaints and ground-truth diagnoses. Structured data was sourced from DiseaseKG, an open-source Chinese medical knowledge graph, while medical textbooks and Wikipedia provided unstructured knowledge.

Key Results and Analysis

ClinicalRAG was tested against several mainstream LLMs, including GPT-3.5, GPT-4, and Llama-2 variants. The results demonstrate a consistent performance boost across all models compared to direct classification.

Diagnostic accuracy comparison between direct classification and the ClinicalRAG pipeline across various models.

Performance Gains

As shown in the results, GPT-4's diagnostic accuracy rose from 82.78% to 84.94% when using the ClinicalRAG pipeline. The system also outperformed specialized medical models like MedKPL (77.78%) and standard BERT-based classifiers (74.07%).

The Importance of Natural Language Synthesis

A key insight from the ablation studies was the criticality of the KNL agent. Converting heterogeneous data into natural language before prompt fusion was the most significant factor in performance. Removing this component led to a 1.53% drop in accuracy, confirming that natural language serves as a "universal interface" for LLM reasoning.

Conclusion and Future Work

ClinicalRAG effectively reduces hallucinations by providing medical-knowledge-aware context. While the multi-agent approach improves accuracy and verifiability, it introduces computational overhead and higher latency. Future work will focus on optimizing these processes for real-time clinical environments and expanding the framework's validation across different languages and diverse Electronic Health Record (EHR) systems.

Summary updated

BibTeX
@misc{lu2024clinicalrag,
  author = {Yuxing Lu and Xukai Zhao and Jinzhuo Wang},
  title = {{ClinicalRAG: Enhancing Clinical Decision Support through Heterogeneous Knowledge Retrieval}},
  year = {2024},
  doi = {10.18653/v1/2024.knowllm-1.6},
}
Figure