Forecasting the publication and citation outcomes of COVID-19 preprints

Michael Gordon, Michaël Bishop, Yiling Chen, Anna Dreber, Brandon Goldfedder, Felix Holzmeister, Magnus Johannesson, Yang Liu, Louisa Tran, Charles Twardy, Juntao Wang, Thomas PfeifferRoyal 2022 · DOI

TL;DR

This study explores whether human forecasters can accurately predict the future publication and citation success of COVID-19 preprints. The findings suggest that crowdsourced forecasts can indeed provide a reliable early assessment of a preprint's potential impact, offering a faster alternative to traditional peer review.

What Problem Does This Paper Address?

The COVID-19 pandemic saw an unprecedented surge in scientific research, with many findings disseminated rapidly through preprint servers like medRxiv and bioRxiv. While preprints accelerate knowledge sharing, their reliability and eventual publication in peer-reviewed journals remain uncertain. This creates a challenge for policymakers, researchers, and the public in discerning the quality and potential impact of this rapidly growing body of work. The paper aims to investigate if human forecasters can bridge this gap by predicting which preprints will be published and how widely they will be cited.

Methods

The researchers employed a crowdsourced forecasting approach. They recruited participants, many from the Systematizing Confidence in Open Research and Evidence (SCORE) project, and presented them with abstracts and links to 400 COVID-19 preprints. Participants were asked to forecast:

  • The probability of a preprint being published in a peer-reviewed journal within one year (categorized by journal impact factor: <10 or >=10).
  • The preprint's relative 1-year Google Scholar citation rank compared to others in the study (on a scale of 0-100).
  • Subjective assessments of the preprint's agreement with future findings and its usefulness in mitigating the pandemic.

These forecasts were incentivized using a surrogate scoring rule (SSR) to encourage truthful and accurate predictions. The actual publication outcomes and citation counts were then resolved approximately one year after the preprints were uploaded.

Datasets

The study utilized a dataset of 400 COVID-19 preprints from medRxiv and bioRxiv, uploaded before the end of August 2020. These preprints were selected based on their Altmetric scores to ensure they represented widely disseminated research. The publication outcomes and Google Scholar citation counts were collected approximately one year after the preprint's initial upload.

Key Insights and Ideas

  • Human intuition is surprisingly effective: The study demonstrates that ordinary individuals, when provided with information, can make remarkably accurate predictions about the future trajectory of scientific research.
  • Preprints are a mixed bag, but many find a home: While not all preprints are destined for publication, a significant majority (70%) of those in this sample were published within a year, with a substantial portion (45%) appearing in high-impact journals.
  • Forecasting offers a speed advantage: Human forecasts provide a preliminary assessment of a preprint's potential much faster than the traditional peer-review process.
  • Subjective assessments correlate with objective outcomes: Forecasters' opinions on a preprint's agreement with future findings and its usefulness were positively correlated with actual publication and citation success.

Key Results

  • Publication Prediction: Forecasters were able to predict publication outcomes with statistically significant positive correlations. Their accuracy in predicting whether a preprint would be published was 59% for preprints with unknown outcomes at the time of the survey.
  • High-Impact Journal Prediction: Forecasts for publication in high-impact journals also showed a significant correlation with the actual outcome (r = 0.38).
  • Citation Prediction: Mean forecasts for citation ranks were highly correlated with actual citation ranks (r = 0.75), indicating strong predictive power for citation impact.
  • Publication Rate: 70% of the sampled preprints were published within one year of their initial upload. Of these, 45% appeared in high-impact journals (JIF >= 10).
  • Citation Volume: On average, preprints received 162 citations within their first year.

Limitations and Future Work

The authors acknowledge several limitations:

  • Sampling Bias: The use of Altmetric scores for sampling might have skewed the results towards preprints that already had significant attention, potentially leading to a higher publication rate than in a random sample.
  • Information Availability: Participants in the forecasting survey might have had access to some information about publication outcomes, especially for preprints uploaded earlier in the study period.
  • Subjectivity: While subjective assessments correlated with objective outcomes, their accuracy in terms of "truth" is difficult to determine, and they may be influenced by factors unrelated to methodological quality.

Future work could explore:

  • Investigating whether these forecasting methods can identify specific methodological problems in preprints.
  • Applying this forecasting approach to other scientific fields beyond COVID-19.
  • Conducting further studies with forecasting occurring immediately after preprint release to minimize information leakage.
  • Exploring the use of prediction markets for forecasting preprint outcomes.

Summary updated

Abstract

Many publications on COVID-19 were released on preprint servers such as <i>medRxiv</i> and <i>bioRxiv.</i> It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high <i>Altmetric</i> score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to <i>Google Scholar</i> citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.
BibTeX
@inproceedings{gordon2022forecasting,
  author = {Michael Gordon and Michaël Bishop and Yiling Chen and Anna Dreber and Brandon Goldfedder and Felix Holzmeister and Magnus Johannesson and Yang Liu and Louisa Tran and Charles Twardy and Juntao Wang and Thomas Pfeiffer},
  title = {{Forecasting the publication and citation outcomes of COVID-19 preprints}},
  booktitle = {Royal},
  year = {2022},
  doi = {10.1098/rsos.220440},
}
Figure