Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations

Rui Yang, Jiayi Tong, Haoyuan Wang, Hui Huang, Ziyang Hu, Peiyu Li, Nan Liu, Christopher J. Lindsell, Michael J. Pencina, Yong Chen, Chuan Hong

Introduction

In medical research, systematic reviews represent the pinnacle of evidence-based practice. However, the traditional peer-review process introduces significant delays, often taking months or years for critical findings to reach the public domain. While preprints (e.g., medRxiv) offer immediate access to data—a necessity during global health crises or in fast-moving fields like cardiovascular disease (CVD)—they are often excluded from systematic reviews due to concerns over methodological rigor and the labor-intensive nature of manual vetting.

This paper introduces AutoConfidence, an automated framework designed to assess the quality of medical preprints using Large Language Models (LLMs) and semantic embeddings. By predicting the likelihood of a preprint's eventual publication in high-impact journals, AutoConfidence enables researchers to include cutting-edge findings in evidence synthesis without compromising scientific integrity.

The AutoConfidence Framework

The AutoConfidence pipeline consists of four primary stages designed to move beyond simple metadata and analyze the substantive content of research papers.

The AutoConfidence framework: a multi-stage pipeline for automated data extraction, semantic embedding, LLM evaluation, and predictive modeling.
  1. Automated Data Extraction: The system utilizes an NLP pipeline to scrape metadata and usage metrics (such as abstract and PDF views) from medRxiv. It tracks the eventual publication status of these preprints via PubMed to establish ground truth.
  2. Semantic Embeddings: Titles and abstracts are processed using MedCPT, a specialized transformer model for biomedical text. These high-dimensional vectors are then compressed using UMAP to capture the thematic essence of the research.
  3. LLM-Driven Evaluation: GPT-4o is prompted to act as a "stringent peer reviewer," scoring preprints on a scale of 1–10 across five dimensions: Originality, Significance, Quality of Presentation, Depth of Research, and Interest to Readers.
  4. Predictive Modeling: The extracted features are fed into a Random Forest Classifier for binary publication prediction and a Survival Cure Model to estimate the time-to-publication risk.

Experimental Setup and Datasets

The study focused on Cardiovascular Diseases (CVD), analyzing a final sample of 1,083 articles (385 published, 698 unpublished) from an initial pool of 2,162 medRxiv preprints. "High-quality" publication was strictly defined as acceptance into a Q1 journal (top 25% in its field) with an Impact Factor ≥ 5.

Key Results and Insights

The integrated approach demonstrated robust predictive capabilities:

  • Binary Prediction: The Random Forest model achieved an AUROC of 0.747 by combining LLM scores, semantic embeddings, and usage metrics.
  • Time-to-Publication: The survival cure model reached an AUROC of 0.731 for predicting eventual publication and a C-index of 0.667 for estimating publication timing.
  • The "Depth" Differentiator: Statistical analysis revealed that "Depth of Research" was the most significant differentiator between published (7.42/10) and unpublished (6.31/10) articles ($p < 0.001$). This suggests that LLMs are particularly adept at identifying the comprehensiveness of an analysis.

A key finding was the synergy between feature types. While GPT-4o alone achieved 68.3% accuracy, its performance as a feature within a traditional machine learning model significantly enhanced the overall predictive power.

Discussion and Limitations

While promising, the AutoConfidence framework has limitations. The current model is domain-specific to cardiovascular research, and the "vibe" of high-quality research may vary across fields like genomics or psychology. Furthermore, using journal prestige as a proxy for quality is a conservative benchmark that may overlook solid research in specialized journals. Future work aims to incorporate full-text semantic embeddings and expand the framework to other medical domains.

Conclusion

AutoConfidence represents a significant step toward making systematic reviews more inclusive and timely. By automating the confidence assessment of preprints, the framework provides a scalable solution for integrating the latest scientific evidence into clinical and policy decision-making processes.

Summary updated

BibTeX
@article{yang2025enabling,
  author = {Rui Yang and Jiayi Tong and Haoyuan Wang and Hui Huang and Ziyang Hu and Peiyu Li and Nan Liu and Christopher J. Lindsell and Michael J. Pencina and Yong Chen and Chuan Hong},
  title = {{Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2025},
  doi = {10.1093/jamia/ocaf137},
}
Figure