ClinicalRAG: Automating Pharmaceutical Label Quality Control with Hierarchical RAG and Large Language Models

Qiaohui Zhou, Zhongliang Zhou, Michael Johnson, Michelle Ngo, Federico Ferrari, Junshui MaAAAI 2025 · DOI

Abstract

The verification of pharmaceutical drug labels against clinical trial data is a critical, labor-intensive process. We present ClinicalRAG, a specialized hierarchical Retrieval-Augmented Generation (RAG) pipeline designed to automate this quality control (QC) process. By organizing clinical trial data into a structured hierarchy and employing a multi-stage verification process, ClinicalRAG achieves high accuracy and specificity, significantly reducing hallucination risks and retrieval failures common in standard large language model (LLM) applications.

Introduction

Before a new medicine reaches the market, every claim on its label—from dosage instructions to side effect frequencies—must be meticulously verified against thousands of tables, listings, and figures (TLFs) generated during clinical trials. This manual process is performed by domain experts and can take months, potentially delaying patient access to life-saving treatments.

While LLMs offer potential for automation, they suffer from hallucinations and retrieval failures in this context. Standard RAG systems often struggle with "flat" data structures where multiple tables (e.g., demographic tables for different study phases) appear semantically similar, leading search algorithms to retrieve incorrect data sources.

Methods

ClinicalRAG addresses these challenges by moving away from flat document chunking in favor of a hierarchical approach. The pipeline consists of four main modules:

  1. Hierarchical Database Construction: Clinical trial tables (originally in RTF format) are converted to Markdown and organized into a tree structure based on Compound, Indication, and Protocol.
  2. Query Synthesizer: An LLM breaks label paragraphs into discrete, atomic statements and extracts metadata (e.g., study ID, drug name) to create structured queries.
  3. Hierarchical Database Retriever: The system performs a Depth-First Search (DFS) through the database tree using the structured query, ensuring the model identifies the correct study's tables before applying an LLM-based reranker.
  4. Self-Consistency Verifier: To ensure numerical accuracy, the system uses a "TableQA" approach. It converts label statements into specific questions, extracts answers from the retrieved tables, and compares them to the original label claims.
Overview of the ClinicalRAG pipeline architecture, including database construction, query synthesis, and verification modules.

Experimental Setup

The system was evaluated using the official KEYTRUDA® label from MSD. A ground truth dataset of 68 quantitative results was verified by expert statisticians. To test the system's ability to identify errors (specificity), the authors used GPT-4 to generate an equal number of synthetic false statements.

Key Results

ClinicalRAG significantly outperformed both "Vanilla" RAG and raw LLM (GPT-4) solutions across all primary metrics.

Performance comparison of ClinicalRAG against GPT-4 and Vanilla RAG across sensitivity, specificity, and accuracy metrics.

Key findings include:

  • High Accuracy: The system achieved 96.1% accuracy in internal validation.
  • Superior Specificity: In Top-1 retrieval scenarios, ClinicalRAG achieved 100% specificity, meaning it never incorrectly verified a false statement.
  • Robustness to Information Overload: While GPT-4's accuracy dropped to 59% when provided with all tables due to confusion between similar data sources, ClinicalRAG’s hierarchical filtering effectively solved the "lost in the middle" problem.

Discussion and Conclusion

The success of ClinicalRAG demonstrates that leveraging existing organizational hierarchies in R&D data is more effective than simple vector-based semantic searches in specialized domains. Furthermore, the "verification via questioning" (TableQA) approach reduces the "yes-man" bias often observed in LLMs.

Future work will focus on expanding the pipeline to handle more complex table layouts and non-standard formats, as well as testing the system across a wider variety of therapeutic areas and document types, such as clinical study reports (CSRs).

Summary updated

BibTeX
@inproceedings{zhou2025clinicalrag,
  author = {Qiaohui Zhou and Zhongliang Zhou and Michael Johnson and Michelle Ngo and Federico Ferrari and Junshui Ma},
  title = {{ClinicalRAG: Automating Pharmaceutical Label Quality Control with Hierarchical RAG and Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35384},
}
Figure