The collective wisdom in the COVID-19 research: Comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles

Yuejiao Wang, Zhidong Cao, Daniel Zeng, Qingpeng Zhang, Tianyi Luo

Abstract

During the initial stages of the COVID-19 pandemic, the scientific community faced an unprecedented "infodemic," characterized by a rapid surge in research papers, many of which were released as non-peer-reviewed preprints. This study evaluates the academic value of these preprints by comparing their epidemiological parameter estimates—specifically the basic reproduction number ($R_0$), incubation period, infectious period, and case-fatality-rate (CFR)—with those found in peer-reviewed literature. Analyzing 106 papers related to the outbreak in China, the research finds that preprint estimates were largely consistent with peer-reviewed findings while offering a significant advantage in timeliness. By synthesizing these data points through bootstrap methods and validating them via a Susceptible-Exposed-Infectious-Recovered-Dead-Cumulative (SEIRDC) model, this study demonstrates that the collective wisdom of preprints is a vital, reliable resource for real-time epidemic decision-making.

Introduction

When a novel pathogen emerges, the speed of scientific communication becomes a matter of life and death. Policymakers and healthcare providers rely on four critical epidemiological parameters to design interventions: the basic reproduction number ($R_0$), the incubation period, the infectious period, and the case-fatality-rate (CFR). Historically, the "gold standard" for such data has been peer-reviewed journals. However, the traditional peer-review process is often too slow to keep pace with a fast-moving pandemic.

This delay led to the rise of preprint servers like medRxiv and bioRxiv, which allowed for the immediate dissemination of findings. Yet, this shift was met with skepticism; many feared that the lack of formal review would lead to the spread of inaccurate or low-quality data. This paper investigates whether that skepticism was justified by quantitatively comparing preprints and peer-reviewed articles and exploring the potential of synthesizing both to reduce uncertainty.

Methodology

The researchers conducted a systematic review of literature published by March 20, 2020, covering databases such as PubMed, Google Scholar, medRxiv, bioRxiv, arRxiv, and SSRN. The focus was limited to studies estimating the four key parameters within the context of the Chinese outbreak.

To analyze the data, the study employed three primary techniques:

  1. Statistical Comparison: The researchers compared the distribution of parameter estimates and the "publication delay" ($T_D$)—the time between the study's completion and its public availability—between preprints and peer-reviewed papers.
  2. Bootstrap Synthesis: A bootstrap resampling method was used to generate robust mean estimates and 95% confidence intervals from the combined pool of all available literature.
  3. SEIRDC Modeling: To verify the real-world applicability of these synthesized parameters, they were integrated into a Susceptible-Exposed-Infectious-Recovered-Dead-Cumulative (SEIRDC) model. The model's output was then compared against actual cumulative case data from China.

Comparative Analysis: Preprints vs. Peer Review

Of the 106 papers included in the final analysis, preprints dominated the landscape, accounting for 72% to 85% of the literature depending on the parameter.

The Timeliness Gap

The most striking difference was the speed of publication. Preprints typically featured a publication delay of only 1–2 days. In contrast, peer-reviewed journals required anywhere from 5 to 16 days to publish. In an escalating crisis, this two-week gap represents a critical window where policy decisions must be made with the best available evidence.

Quality and Consistency

Contrary to fears of "low-quality" preprint data, the study found that the distributions for $R_0$, incubation period, and infectious period were nearly identical across both groups. The only notable discrepancy occurred in the Case Fatality Rate (CFR). Peer-reviewed papers reported a higher mean CFR (6.10%) than preprints (4.26%). The authors attribute this to a selection bias in early high-impact journals, which often prioritized studies of the most severe initial clinical cases, whereas preprints captured a broader, more evolving data set as the epidemic progressed.

Synthesized Results and Validation

By pooling the "collective wisdom" of the entire research community, the study produced the following synthesized estimates for the COVID-19 outbreak in China:

  • Basic Reproduction Number ($R_0$): 3.18 (95% CI 2.85–3.53)
  • Incubation Period: 5.44 days (95% CI 4.98–5.99)
  • Infectious Period: 6.25 days (95% CI 5.09–7.51)
  • Case-Fatality-Rate (CFR): 4.51% (95% CI 3.41%–6.29%)

To validate these figures, the researchers plugged them into the SEIRDC model. The resulting simulated infection curve matched the real-world onset cases in China with remarkable accuracy, particularly during the early stages of the outbreak. This suggests that the synthesized parameters effectively captured the underlying dynamics of the virus.

Discussion and Implications

The findings suggest that the academic value of preprints has been significantly underestimated. Rather than being a source of noise, preprints acted as a sensitive, real-time reflection of the epidemic's trajectory. While the results of any single preprint should be viewed with caution, the aggregate data from preprint servers provides a robust and timely foundation for epidemiological modeling.

However, the study notes certain limitations. The data was geographically specific to China, and parameters like CFR and infectious period are dynamic, changing as medical interventions and testing capacities improve. Furthermore, the study highlights the need for automated tools to synthesize the massive volume of research generated during health crises.

Conclusion

The scientific community should actively capitalize on the collective intelligence found in preprints. By synthesizing early-stage research from both peer-reviewed and preprint sources, researchers and policymakers can reduce uncertainty and make more informed decisions during the critical early windows of emerging infectious diseases.

Summary updated

Abstract

The validity of the COVID-19 parameter estimations of the preprints was on par with that of peer-reviewed publications, and synthesized results of literatures could reduce the uncertainty and be used for epidemic decision-making.
BibTeX
@article{wang2020collective,
  author = {Yuejiao Wang and Zhidong Cao and Daniel Zeng and Qingpeng Zhang and Tianyi Luo},
  title = {{The collective wisdom in the COVID-19 research: Comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles}},
  journal = {International Journal of Infectious Diseases},
  year = {2020},
  doi = {10.1016/j.ijid.2020.12.040},
}
Figure