The collective wisdom in the COVID-19 research: comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles

Yuejiao Wang, Zhidong Cao, Daniel Zeng, Qingpeng Zhang, Tianyi Luo

Abstract

During the initial stages of the COVID-19 pandemic, the scientific community faced an unprecedented "infodemic," characterized by a rapid surge in research papers, many of which were released as non-peer-reviewed preprints. This study evaluates the academic value of these preprints by comparing their epidemiological parameter estimates—specifically the basic reproduction number ($R_0$), incubation period, infectious period, and case-fatality-rate (CFR)—with those found in peer-reviewed literature. Analyzing 106 papers related to the outbreak in China, the research finds that preprint estimates were largely consistent with peer-reviewed findings while offering a significant advantage in timeliness. By synthesizing these data points through bootstrap methods and validating them via a Susceptible-Exposed-Infectious-Recovered-Dead-Cumulative (SEIRDC) model, this study demonstrates that the collective wisdom of preprints is a vital, reliable resource for real-time epidemic decision-making.

Introduction

When a novel pathogen emerges, the speed of scientific communication becomes a matter of life and death. Policymakers and healthcare providers rely on four critical epidemiological parameters to design interventions: the basic reproduction number ($R_0$), the incubation period, the infectious period, and the case-fatality-rate (CFR). Historically, the "gold standard" for such data has been peer-reviewed journals. However, the traditional peer-review process is often too slow to keep pace with a fast-moving pandemic.

This delay led to the rise of preprint servers like medRxiv and bioRxiv, which allowed for the immediate dissemination of findings. Yet, this shift was met with skepticism; many feared that the lack of formal review would lead to the spread of inaccurate or low-quality data. This paper investigates whether that skepticism was justified by quantitatively comparing preprints and peer-reviewed articles and exploring the potential of synthesizing both to reduce uncertainty.

Methodology

The researchers conducted a systematic review of literature published by March 20, 2020, covering databases such as PubMed, Google Scholar, medRxiv, bioRxiv, arRxiv, and SSRN. The focus was limited to studies estimating the four key parameters within the context of the Chinese outbreak.

To analyze the data, the study employed three primary techniques:

  1. Statistical Comparison: The researchers compared the distribution of parameter estimates and the "publication delay" ($T_D$)—the time between the study's completion and its public availability—between preprints and peer-reviewed papers.
  2. Bootstrap Synthesis: A bootstrap resampling method was used to generate robust mean estimates and 95% confidence intervals from the combined pool of all available literature.
  3. SEIRDC Modeling: To verify the real-world applicability of these synthesized parameters, they were integrated into a Susceptible-Exposed-Infectious-Recovered-Dead-Cumulative (SEIRDC) model. The model's output was then compared against actual cumulative case data from China.

Comparative Analysis: Preprints vs. Peer Review

Of the 106 papers included in the final analysis, preprints dominated the landscape, accounting for 72% to 85% of the literature depending on the parameter.

The Timeliness Gap

The most striking difference was the speed of publication. Preprints typically featured a publication delay of only 1–2 days. In contrast, peer-reviewed journals required anywhere from 5 to 16 days to publish. In an escalating crisis, this two-week gap represents a critical window where policy decisions must be made with the best available evidence.

Quality and Consistency

Contrary to fears of "low-quality" preprint data, the study found that the distributions for $R_0$, incubation period, and infectious period were nearly identical across both groups. The only notable discrepancy occurred in the Case Fatality Rate (CFR). Peer-reviewed papers reported a higher mean CFR (6.10%) than preprints (4.26%). The authors attribute this to a selection bias in early high-impact journals, which often prioritized studies of the most severe initial clinical cases, whereas preprints captured a broader, more evolving data set as the epidemic progressed.

Synthesized Results and Validation

By pooling the "collective wisdom" of the entire research community, the study produced the following synthesized estimates for the COVID-19 outbreak in China:

  • Basic Reproduction Number ($R_0$): 3.18 (95% CI 2.85–3.53)
  • Incubation Period: 5.44 days (95% CI 4.98–5.99)
  • Infectious Period: 6.25 days (95% CI 5.09–7.51)
  • Case-Fatality-Rate (CFR): 4.51% (95% CI 3.41%–6.29%)

To validate these figures, the researchers plugged them into the SEIRDC model. The resulting simulated infection curve matched the real-world onset cases in China with remarkable accuracy, particularly during the early stages of the outbreak. This suggests that the synthesized parameters effectively captured the underlying dynamics of the virus.

Discussion and Implications

The findings suggest that the academic value of preprints has been significantly underestimated. Rather than being a source of noise, preprints acted as a sensitive, real-time reflection of the epidemic's trajectory. While the results of any single preprint should be viewed with caution, the aggregate data from preprint servers provides a robust and timely foundation for epidemiological modeling.

However, the study notes certain limitations. The data was geographically specific to China, and parameters like CFR and infectious period are dynamic, changing as medical interventions and testing capacities improve. Furthermore, the study highlights the need for automated tools to synthesize the massive volume of research generated during health crises.

Conclusion

The scientific community should actively capitalize on the collective intelligence found in preprints. By synthesizing early-stage research from both peer-reviewed and preprint sources, researchers and policymakers can reduce uncertainty and make more informed decisions during the critical early windows of emerging infectious diseases.

Summary updated

Abstract

Summary Background Research papers related to COVID-19 have exploded. We aimed to explore the academic value of preprints through comparing with peer-reviewed publications, and synthesize the parameter estimates of the two kinds of literature. Method We collected papers regarding the estimation of four key epidemiological parameters of the COVID-19 in China: the basic reproduction number (R 0 ), incubation period, infectious period, and case-fatality-rate (CFR). PubMed, Google Scholar, medRxiv, bioRxiv, arRxiv, and SSRN were searched by 20 March, 2020. Distributions of parameters and timeliness of preprints and peer-reviewed papers were compared. Further, four parameters were synthesized by bootstrap, and their validity was verified by susceptible-exposed-infectious-recovered-dead-cumulative (SEIRDC) model based on the context of China. Findings 106 papers were included for analysis. The distributions of four parameters in two literature groups were close, despite that the timeliness of preprints was better. Four parameter estimates changed over time. Synthesized estimates of R 0 (3·18, 95% CI 2·85-3·53), incubation period (5·44 days, 95% CI 4·98-5·99), infectious period (6·25 days, 95% CI 5·09-7·51), and CFR (4·51%, 95% CI 3·41%-6·29%) were obtained from the whole parameters space, all with p<0·05. Their validity was evaluated by simulated cumulative cases of SEIRDC model, which matched well with the onset cases in China. Interpretation Preprints could reflect the changes of epidemic situation sensitively, and their academic value shouldn’t be neglected. Synthesized results of literatures could reduce the uncertainty and be used for epidemic decision making. Funding The National Natural Science Foundation of China and Beijing Municipal Natural Science Foundation. Research in context Evidence before this study Since its outbreak, scientific articles about the COVID-19 have greatly surged, with a significant portion as non-peer-reviewed preprints. Although preprints captured great attention, the credibility of preprints was widely debated. We searched PubMed and Google on March 20, 2020, for publications that discussed the preprints during the COVID-19 pandemic, using the terms (“preprints” AND “COVID-19”). We identified 12 papers and news, and found that scientists were skeptical of preprints mainly because rigorous peer review is absent and thus the conclusions of preprints may not be reliable. However, scientists’ opinions could have been biased towards limited data, and there is few knowledges about the validity of the results reported in the preprints. Further, to examine how scientists utilize results of preprints, taking the epidemiological parameter estimation as the objects, we searched reviews on Google using the terms (“epidemiology” AND (“meta-analysis” OR “reviews”) AND “COVID-19”) on May 23, 2020. Nine papers were identified. We found that existing meta-analysis and reviews included few preprints. This may be due to the fact that the quality of preprints was not recognized, and thus their academic value was underestimated. Overall, the validity of the results as reported in the preprints should be further examined and the potential of synthesizing preprints with formally published papers should be explored. Added value of this study Our study adds value in four main ways. First, we collected preprints and peer-reviewed papers on estimations of the four most important epidemiological parameters (the basic reproduction number, incubation period, infectious period, and case-fatality-rate) for the COVID-19 outbreak in China. 106 papers were included and available data were extracted. Second, we quantitatively compared the differences and timeliness between preprints and peer-reviewed publications in the estimation of the four parameters, and found that the validity of the preprints’ estimations was largely consistent with that of the peer-reviewed group. Third, we synthesized the estimations of the two groups of literatures using bootstrap method, and found that the values of infectious period and case-fatality-rate decreased over time, indicating that the synthesized results timely reflected the changing trend of the COVID-19 in China. Finally, the practicability of the synthesized parameter estimations was verified by the data of confirmed cases in China. The cumulative infection curve simulated using synthesized parameters fitted the real data well. Implications of all the available evidence Results of our study indicate that the validity of the COVID-19 parameter estimations of the preprints is on par with that of peer-reviewed publications, and the preprints are relatively timelier. Further, the synthesized parameters of the two literature groups can effectively reduce the uncertainty and capture the patterns of epidemics. These results provide data-driven insights into the academic value of preprints, which have been arguably underestimated. The scientific community should actively capitalize the collective wisdom generated by the huge amount of preprints, particularly during the emerging infectious diseases like the COVID-19.
BibTeX
@article{wang2020collective,
  author = {Yuejiao Wang and Zhidong Cao and Daniel Zeng and Qingpeng Zhang and Tianyi Luo},
  title = {{The collective wisdom in the COVID-19 research: comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles}},
  journal = {International Journal of Infectious Diseases},
  year = {2020},
  doi = {10.1016/j.ijid.2020.12.040},
}
Figure