TL;DR
This paper introduces RAPID, an automated tool that uses large language models (LLMs) and retrieval augmentation to generate medical reporting checklists for clinical trial manuscripts. RAPID significantly improves efficiency and accuracy in assessing adherence to reporting guidelines, offering a scalable solution for authors, researchers, and editors.
What Problem Does This Paper Address?
Medical research, especially randomized clinical trials (RCTs), relies heavily on reporting guidelines to ensure transparency, quality, and interpretability. However, with over 600 such guidelines, manually checking manuscripts for compliance is a time-consuming and labor-intensive task. Existing automated tools have limitations, such as incomplete item coverage, lack of transparency in their judgments, and inflexibility requiring retraining for new guidelines. This paper aims to address these challenges by developing a more robust, adaptable, and transparent automated system.
Methods
The core of RAPID is a retrieval augmentation generation (RAG) framework powered by LLMs. The process involves several key steps:
- Input Preparation: Medical papers (in PDF or Word format) are preprocessed, including text segmentation and medical reporting guideline preprocessing.
- Agent Framework: This is where the LLMs come into play.
- Paragraph Retrieval: LLMs identify paragraphs relevant to specific checklist items by breaking down complex items into simpler sub-queries. This "item decomposition" enhances accuracy.
- Iterative Paragraph Selection: A multi-round strategy selects relevant paragraphs, ensuring context is maintained and redundancy is reduced.
- Information Judgment: LLMs evaluate the retrieved paragraphs to determine if an item is "Reported" or "Not Reported," providing explanations for their judgments.
- Summarization: Finally, the LLM summarizes the findings to generate the checklist report.
RAPID also incorporates "structured prompts" and "output restrictions" (like JSON format) to improve consistency and facilitate automated processing. A secondary correction mechanism is used to mitigate potential LLM errors.
Datasets
The study utilized two datasets for evaluation:
- CONSORT Dataset: 50 randomized controlled trials (RCTs) published within the last five years, selected from PubMed.
- CONSORT-AI Dataset: 41 RCTs specifically involving AI interventions, sourced from a previous systematic review.
These datasets were manually annotated by researchers to serve as the "gold standard" for evaluating RAPID's performance.
Key Insights and Ideas
- Decomposition is Key: Breaking down complex checklist items into simpler sub-queries significantly improves the LLM's ability to pinpoint relevant information.
- RAG Enhances Transparency: By retrieving specific paragraphs and providing explanations, RAPID offers more insight into its judgments compared to "black box" NLP models.
- Adaptability Without Retraining: RAPID's architecture allows it to be adapted to different medical reporting guidelines without the need for extensive retraining on new datasets, a significant advantage over previous methods.
- Open-Source LLMs Shine: Integrating RAPID with open-source LLMs can achieve performance comparable to or even exceeding that of top-tier closed-source models, addressing concerns about proprietary model usage.
Key Results
RAPID demonstrated strong performance on both datasets:
- CONSORT Dataset: Achieved an average accuracy of 92.11% and a content consistency score (CCS) of 81.14% when combined with the deepseek-chat LLM.
- CONSORT-AI Dataset: Achieved an average accuracy of 83.81% and a CCS of 72.51% when combined with the deepseek-chat LLM.
These results were competitive with, and often superior to, various other LLMs (including ChatGPT-4, ChatGPT-3.5, and Claude-3.5-Sonnet) when used with a "Vanilla Prompt" method. RAPID also showed robust performance across different LLMs, highlighting its scalability and extensibility.
Limitations and Future Work
The authors acknowledge several limitations:
- Prompt Optimization: The prompts were specifically optimized for ChatGPT, which might not be ideal for all LLMs.
- Token Consumption: The architecture has relatively high token consumption, suggesting room for optimization.
- LLM Variability: Performance variability was observed across different LLMs when summarizing final answers, particularly for certain items. This can be partially mitigated by secondary correction or model ensembling.
Future work will focus on continuously optimizing the architecture, reducing token usage, and extending RAPID to cover a wider range of medical reporting guidelines. The development of a user-friendly web application is also a key outcome, providing a streamlined interface for accessing RAPID's capabilities.
Summary updated