RAPID: Reliable and efficient Automatic generation of submission rePorting checklists with Large language moDels

Zeming Li, Xufei Luo, Zhenhua Yang, Huayu Zhang, Bingyi Wang, Long Ge, Zhaoxiang Bian, James Zou, Yaolong Chen, Lu ZhangbioRxiv 2025 · DOI

TL;DR

This paper introduces RAPID, an automated tool that uses large language models (LLMs) and retrieval augmentation to generate medical reporting checklists for clinical trial manuscripts. RAPID significantly improves efficiency and accuracy in assessing adherence to reporting guidelines, offering a scalable solution for authors, researchers, and editors.

What Problem Does This Paper Address?

Medical research, especially randomized clinical trials (RCTs), relies heavily on reporting guidelines to ensure transparency, quality, and interpretability. However, with over 600 such guidelines, manually checking manuscripts for compliance is a time-consuming and labor-intensive task. Existing automated tools have limitations, such as incomplete item coverage, lack of transparency in their judgments, and inflexibility requiring retraining for new guidelines. This paper aims to address these challenges by developing a more robust, adaptable, and transparent automated system.

Methods

The core of RAPID is a retrieval augmentation generation (RAG) framework powered by LLMs. The process involves several key steps:

  • Input Preparation: Medical papers (in PDF or Word format) are preprocessed, including text segmentation and medical reporting guideline preprocessing.
  • Agent Framework: This is where the LLMs come into play.
  • Paragraph Retrieval: LLMs identify paragraphs relevant to specific checklist items by breaking down complex items into simpler sub-queries. This "item decomposition" enhances accuracy.
  • Iterative Paragraph Selection: A multi-round strategy selects relevant paragraphs, ensuring context is maintained and redundancy is reduced.
  • Information Judgment: LLMs evaluate the retrieved paragraphs to determine if an item is "Reported" or "Not Reported," providing explanations for their judgments.
  • Summarization: Finally, the LLM summarizes the findings to generate the checklist report.

RAPID also incorporates "structured prompts" and "output restrictions" (like JSON format) to improve consistency and facilitate automated processing. A secondary correction mechanism is used to mitigate potential LLM errors.

Datasets

The study utilized two datasets for evaluation:

  • CONSORT Dataset: 50 randomized controlled trials (RCTs) published within the last five years, selected from PubMed.
  • CONSORT-AI Dataset: 41 RCTs specifically involving AI interventions, sourced from a previous systematic review.

These datasets were manually annotated by researchers to serve as the "gold standard" for evaluating RAPID's performance.

Key Insights and Ideas

  • Decomposition is Key: Breaking down complex checklist items into simpler sub-queries significantly improves the LLM's ability to pinpoint relevant information.
  • RAG Enhances Transparency: By retrieving specific paragraphs and providing explanations, RAPID offers more insight into its judgments compared to "black box" NLP models.
  • Adaptability Without Retraining: RAPID's architecture allows it to be adapted to different medical reporting guidelines without the need for extensive retraining on new datasets, a significant advantage over previous methods.
  • Open-Source LLMs Shine: Integrating RAPID with open-source LLMs can achieve performance comparable to or even exceeding that of top-tier closed-source models, addressing concerns about proprietary model usage.

Key Results

RAPID demonstrated strong performance on both datasets:

  • CONSORT Dataset: Achieved an average accuracy of 92.11% and a content consistency score (CCS) of 81.14% when combined with the deepseek-chat LLM.
  • CONSORT-AI Dataset: Achieved an average accuracy of 83.81% and a CCS of 72.51% when combined with the deepseek-chat LLM.

These results were competitive with, and often superior to, various other LLMs (including ChatGPT-4, ChatGPT-3.5, and Claude-3.5-Sonnet) when used with a "Vanilla Prompt" method. RAPID also showed robust performance across different LLMs, highlighting its scalability and extensibility.

Limitations and Future Work

The authors acknowledge several limitations:

  • Prompt Optimization: The prompts were specifically optimized for ChatGPT, which might not be ideal for all LLMs.
  • Token Consumption: The architecture has relatively high token consumption, suggesting room for optimization.
  • LLM Variability: Performance variability was observed across different LLMs when summarizing final answers, particularly for certain items. This can be partially mitigated by secondary correction or model ensembling.

Future work will focus on continuously optimizing the architecture, reducing token usage, and extending RAPID to cover a wider range of medical reporting guidelines. The development of a user-friendly web application is also a key outcome, providing a streamlined interface for accessing RAPID's capabilities.

Summary updated

Abstract

Abstract Importance Medical reporting guidelines are significant in improving the transparency, quality, and integrity of medical research, particularly in randomized clinical trials; adherence to these guidelines supports research interpretability and has direct implications for downstream applications, such as patient treatment. However, with over 600 distinct reporting guidelines, manual assessments are often time-consuming and labor-intensive. Objective To evaluate an automated reporting checklist generation tool using large language models and retrieval augmentation generation technology, called RAPID. Design, Setting, and Participants This study used large language models to design a retrieval augmentation generation architecture and collected published journal articles as training and validation sets to optimize prompts within the framework and comprehensively evaluate the performance of the framework. Medical reporting experiments were collected from 50 randomized controlled trials without the intervention of AI tools and 41 randomized controlled trials with the intervention of AI tools. Main Outcomes and Measures For effective evaluation of the performance of this tool, a classification accuracy metric (Reported/Not Reported) defined as the number of correct judgments divided by all judgments and a content consistent score metric defined as the number of contents retrieved by the tool that are the same as those retrieved by researchers divided by the total number of judgments were calculated. Results The RAPID tool uses the widely used Word document and Portable Document Format as an input file. Fifty published journal articles without the intervention of AI tools and 41 published journal articles with the intervention of AI tools were collected as CONSORT and CONSORT-AI datasets. All of the CONSORT reporting items (37) were included in the tool. RAPID achieved a high average accuracy rate of 92.11% and a content consistency score of 81.14% on the CONSORT dataset. Of the CONSORT-AI reporting items, 11 items related to the intervention of AI tools were included in the tool. RAPID achieved an average accuracy of 83.81% with a content consistency score of 72.51% on the CONSORT-AI dataset. For these two reporting guidelines, a training set of 5 articles was selected from each dataset to refine the prompts used in the tools for CONSORT and CONSORT-AI reporting checklist. The validation set of the remaining articles was used to assess the performance of the RAPID. The RAPID tool used the Word document and Portable Document Format of the articles as input files. A RAPID graphical user interface was built using JavaScript and Vue. Conclusions and Relevance The RAPID tool is designed to assist in the reporting of various types of trials. RAPID has strong scalability, which can be easily adapted to different medical reporting guidelines without transfer learning on a large dataset. RAPID may effectively save time and improve working efficiency for different user groups, for example, 1) simplifying the submission process and improving report quality by verifying manuscript completeness for medical authors; 2) facilitating evaluation of report quality for medical researchers; 3) expediting manuscript distribution for medical editors; and 4) identifying reporting deficiencies and providing deeper insights for review comments for reviewers. Key Points Question Can large language model tools automatically generate medical reporting checklists for manuscripts of different types of clinic trials? Findings An automated reporting checklist generation tool using large language models and retrieval augmentation generation technology, RAPID, was developed. RAPID was fully evaluated on all items across two separate datasets related to the Consolidated Standards of Reporting Trials (CONSORT) and CONSORT-AI reporting guidelines. In the first dataset corresponding to CONSORT, RAPID achieved an average accuracy of 92.11%, while in the second dataset associated with CONSORT - AI, it reached an average accuracy of 83.81%. Additionally, RAPID is highly scalable. It can be easily and smoothly adapted to different medical reporting guidelines without transfer learning on a large dataset. Meaning RAPID can effectively save time and improve working efficiency for different user groups such as medical authors, researchers, editors, and reviewers.
BibTeX
@misc{li2025rapid,
  author = {Zeming Li and Xufei Luo and Zhenhua Yang and Huayu Zhang and Bingyi Wang and Long Ge and Zhaoxiang Bian and James Zou and Yaolong Chen and Lu Zhang},
  title = {{RAPID: Reliable and efficient Automatic generation of submission rePorting checklists with Large language moDels}},
  year = {2025},
  doi = {10.1101/2025.02.13.638015},
  howpublished = {bioRxiv},
}
Figure