TL;DR
This study explores whether human forecasters can accurately predict the future publication and citation success of COVID-19 preprints. The findings suggest that crowdsourced forecasts can indeed provide a reliable early assessment of a preprint's potential impact, offering a faster alternative to traditional peer review.
What Problem Does This Paper Address?
The COVID-19 pandemic saw an unprecedented surge in scientific research, with many findings disseminated rapidly through preprint servers like medRxiv and bioRxiv. While preprints accelerate knowledge sharing, their reliability and eventual publication in peer-reviewed journals remain uncertain. This creates a challenge for policymakers, researchers, and the public in discerning the quality and potential impact of this rapidly growing body of work. The paper aims to investigate if human forecasters can bridge this gap by predicting which preprints will be published and how widely they will be cited.
Methods
The researchers employed a crowdsourced forecasting approach. They recruited participants, many from the Systematizing Confidence in Open Research and Evidence (SCORE) project, and presented them with abstracts and links to 400 COVID-19 preprints. Participants were asked to forecast:
- The probability of a preprint being published in a peer-reviewed journal within one year (categorized by journal impact factor: <10 or >=10).
- The preprint's relative 1-year Google Scholar citation rank compared to others in the study (on a scale of 0-100).
- Subjective assessments of the preprint's agreement with future findings and its usefulness in mitigating the pandemic.
These forecasts were incentivized using a surrogate scoring rule (SSR) to encourage truthful and accurate predictions. The actual publication outcomes and citation counts were then resolved approximately one year after the preprints were uploaded.
Datasets
The study utilized a dataset of 400 COVID-19 preprints from medRxiv and bioRxiv, uploaded before the end of August 2020. These preprints were selected based on their Altmetric scores to ensure they represented widely disseminated research. The publication outcomes and Google Scholar citation counts were collected approximately one year after the preprint's initial upload.
Key Insights and Ideas
- Human intuition is surprisingly effective: The study demonstrates that ordinary individuals, when provided with information, can make remarkably accurate predictions about the future trajectory of scientific research.
- Preprints are a mixed bag, but many find a home: While not all preprints are destined for publication, a significant majority (70%) of those in this sample were published within a year, with a substantial portion (45%) appearing in high-impact journals.
- Forecasting offers a speed advantage: Human forecasts provide a preliminary assessment of a preprint's potential much faster than the traditional peer-review process.
- Subjective assessments correlate with objective outcomes: Forecasters' opinions on a preprint's agreement with future findings and its usefulness were positively correlated with actual publication and citation success.
Key Results
- Publication Prediction: Forecasters were able to predict publication outcomes with statistically significant positive correlations. Their accuracy in predicting whether a preprint would be published was 59% for preprints with unknown outcomes at the time of the survey.
- High-Impact Journal Prediction: Forecasts for publication in high-impact journals also showed a significant correlation with the actual outcome (r = 0.38).
- Citation Prediction: Mean forecasts for citation ranks were highly correlated with actual citation ranks (r = 0.75), indicating strong predictive power for citation impact.
- Publication Rate: 70% of the sampled preprints were published within one year of their initial upload. Of these, 45% appeared in high-impact journals (JIF >= 10).
- Citation Volume: On average, preprints received 162 citations within their first year.
Limitations and Future Work
The authors acknowledge several limitations:
- Sampling Bias: The use of Altmetric scores for sampling might have skewed the results towards preprints that already had significant attention, potentially leading to a higher publication rate than in a random sample.
- Information Availability: Participants in the forecasting survey might have had access to some information about publication outcomes, especially for preprints uploaded earlier in the study period.
- Subjectivity: While subjective assessments correlated with objective outcomes, their accuracy in terms of "truth" is difficult to determine, and they may be influenced by factors unrelated to methodological quality.
Future work could explore:
- Investigating whether these forecasting methods can identify specific methodological problems in preprints.
- Applying this forecasting approach to other scientific fields beyond COVID-19.
- Conducting further studies with forecasting occurring immediately after preprint release to minimize information leakage.
- Exploring the use of prediction markets for forecasting preprint outcomes.
Summary updated