TL;DR
This paper explores the use of Large Language Models (LLMs) like GPT-4 to automatically assess how well randomized controlled trials (RCTs) involving AI interventions adhere to the CONSORT-AI reporting guidelines. The study found that advanced LLMs, particularly GPT-4 variants, show promising consistency in this task, though they are not yet fully autonomous and require human oversight for complex evaluations.
What Problem Does This Paper Address?
Reporting guidelines are crucial for ensuring the quality and integrity of scientific research. For clinical trials, the CONSORT (Consolidated Standards of Reporting Trials) statement provides a framework. However, with the rise of AI in healthcare, a specialized guideline, CONSORT-AI, was introduced to address the unique reporting needs of AI-based interventions. Manually checking RCTs against these guidelines is time-consuming and resource-intensive. This paper investigates whether LLMs can automate this process, specifically assessing the consistency of AI intervention RCTs with CONSORT-AI.
Methods
The researchers conducted a cross-sectional study using six different LLM models: GPT-4 variants (gpt-4-0125-preview, gpt-4-1106-preview), ChatGPT 3.5 (gpt-3.5-turbo-0125, gpt-3.5-turbo-1106), and Claude 3 models (Opus, Sonnet). They used a previously published systematic review that evaluated 41 AI intervention RCTs against the 11 key items of the CONSORT-AI guidelines as their gold standard.
Prompt engineering was a critical step, involving iterative refinement of prompts to ensure accurate and concise LLM responses. All queries were submitted via an API with a temperature setting of 0 to ensure deterministic outputs. One researcher posed the questions, and another independently verified the responses for validity. They analyzed the Overall Consistency Score (OCS), recall, inter-rater reliability, and content consistency.
Datasets
The study evaluated 41 randomized controlled trials (RCTs) involving AI interventions. These RCTs were sourced from a previously published systematic review. The paper does not introduce a new benchmark dataset but rather uses existing published RCTs and a gold standard derived from a prior systematic review.
Key Insights and Ideas
- LLMs show promise for CONSORT-AI adherence assessment: The study demonstrates that LLMs, especially GPT-4 variants, can effectively evaluate the consistency of AI intervention RCTs with CONSORT-AI guidelines, offering a potential avenue for automating this laborious process.
- Prompt engineering is key: The performance of LLMs is highly dependent on the quality of the prompts. Iterative refinement was crucial to achieve acceptable accuracy.
- Human oversight remains essential: While LLMs perform well, they are not yet fully autonomous. Complex or nuanced items within the CONSORT-AI guidelines can still pose challenges, requiring human expertise for validation and interpretation.
- Variability in LLM performance: Different LLM models and even different versions of the same model exhibit varying levels of performance in assessing CONSORT-AI adherence.
- Specific CONSORT-AI items are more challenging: Item 2 of CONSORT-AI, which deals with input data criteria, received the poorest overall evaluation, suggesting that LLMs struggle with the precise definition and interpretation of data-level criteria.
Key Results
- GPT-4 variants outperformed other models: GPT-4-0125-preview achieved the highest average Overall Consistency Score (OCS), with an average of 86.5% when using the JAMA Network Open authors' evaluation as the gold standard and 81.6% based on the study's own evaluation.
- GPT-3.5 models lagged behind: The GPT-3.5-turbo-0125 model showed the worst average OCS, around 61.9% to 63.0%.
- Item 2 is a weak point: Item 2 of CONSORT-AI ("State the inclusion and exclusion criteria at the level of the input data") had the lowest average OCS across all models, at 48.8%.
- High consistency for certain items: Items 1, 5, 8, and 9 of CONSORT-AI generally received high OCS scores (above 80%) across the models.
- Moderate human agreement: The agreement between human annotators and the LLM-extracted content was moderate (Cohen's κ = 0.65), indicating that human evaluations can also vary.
Limitations and Future Work
- Prompt ambiguity: The study highlights that LLMs can misinterpret prompts, leading to errors. For instance, GPT-4 confused data-level criteria with patient-level criteria in one instance.
- Reporting of AI versions: Many studies report the time of algorithm development rather than the specific version, which LLMs struggle to interpret as a precise report.
- Model evolution: The study used specific versions of LLMs, and their rapid evolution means findings might change with newer versions.
- PDF format limitations: Evaluating documents in PDF format might yield different results than plain text.
- Sample size: While 41 RCTs is a reasonable sample for this emerging field, a larger dataset could improve generalizability.
- Future directions: The authors suggest optimizing prompts for different reporting guidelines and LLMs, exploring automated or semi-automated prompt generation methods, and strengthening human supervision. They also recommend further studies to evaluate LLM consistency with other reporting guidelines beyond CONSORT-AI.
Summary updated