Statistical Issues and Lessons Learned from COVID-19 Clinical Trials with Lopinavir-Ritonavir and Remdesivir

Guosheng Yin, Chenyang Zhang, Huaqing JinmedRxiv 2020 · DOI

Abstract

Since the outbreak of COVID-19, the global medical community has raced to identify effective treatments through randomized clinical trials (RCTs). This paper examines three high-profile trials: one for lopinavir-ritonavir and two for remdesivir. While these trials provided the initial evidence base for pandemic response, several statistical issues in their design and analysis—including unplanned sample size adjustments and the misuse of proportional hazards models—call for a cautious interpretation of their results. By re-analyzing reconstructed data using Restricted Mean Survival Time (RMST) and terminal rate models, we identify how different statistical choices could have altered the perceived efficacy of these treatments and offer lessons for future pandemic research.

Introduction

In the early months of the COVID-19 pandemic, clinical trials were launched at an unprecedented pace. However, the urgency of the crisis often led to compromises in statistical rigor. This paper provides a critical post-mortem of three landmark studies: the LOTUS China trial (lopinavir-ritonavir), the Wang et al. remdesivir trial in Hubei, and the international Beigel et al. remdesivir trial. Through a re-evaluation of the published cumulative incidence curves, we demonstrate how standard metrics like the Hazard Ratio (HR) can be misleading when the underlying assumptions of the model are violated.

Methodological Critiques

The authors identify three primary statistical hurdles that affected the reliability of early COVID-19 trial data:

1. Type I Error Inflation

In the lopinavir-ritonavir trial, the sample size was increased from 160 to 199 patients after initial results failed to reach significance. Because this adjustment was not part of a pre-defined adaptive design, it inflated the Type I error rate (the probability of a false positive) from 0.05 to 0.071, potentially weakening the trial's statistical integrity.

2. Underpowered Studies and Early Termination

The Wang et al. remdesivir trial was forced to stop early due to a lack of eligible patients as the outbreak in Hubei was brought under control. With only 236 of the planned 453 patients enrolled, the study lacked the power to detect a statistically significant difference, leading to a widely reported "negative" result that may have been premature.

3. Violation of Proportional Hazards

Most COVID-19 trials utilized the Hazard Ratio (HR) as the primary measure of efficacy. HR assumes that the relative risk between the treatment and placebo groups remains constant over time. However, in COVID-19 recovery, where recovery curves often cross or intertwine, this assumption is frequently violated. When proportional hazards do not hold, the HR becomes an unreliable and difficult-to-interpret metric.

Re-analysis and Alternative Metrics

To address these issues, the authors applied more robust statistical techniques to the reconstructed trial data:

  • Restricted Mean Survival Time (RMST): A model-free metric that calculates the average time to an event (such as recovery or improvement) within a specific window (e.g., 28 days). RMST is far more robust than HR when recovery rates vary over the course of the study.
  • Terminal (Cure) Rate Model: Standard survival analysis often treats death as "censoring." However, since a deceased patient will never reach the recovery endpoint, the authors argue that death should be treated as a terminal event in a mixture model to provide a clearer picture of clinical outcomes.
  • Bootstrap Projection: For the underpowered Wang et al. trial, bootstrap methods were used to predict whether results would have reached significance if the original target sample size had been met.

Key Findings

Lopinavir-Ritonavir (LOTUS China)

Re-analysis confirmed the original finding of no significant difference in terminal rates (P=.16). Interestingly, while the 14-day Restricted Mean Time to Improvement (RMTI) showed a slight lead for the drug (-0.67 days, P=.02), this advantage vanished by day 28, suggesting any benefit was transient.

Wang et al. (Remdesivir)

While the actual trial reported a non-significant HR of 1.23, bootstrap simulations suggested that if the trial had reached its target of 453 patients, the results likely would have reached statistical significance (P=.02). If the sample size had been doubled, the significance would have been even more pronounced (P=.01).

Beigel et al. (Remdesivir)

The re-analysis confirmed the superiority of remdesivir but provided a more intuitive clinical metric. The RMTR (Restricted Mean Time to Recovery) difference was -2.7 days (P < .001), meaning patients on remdesivir recovered, on average, nearly three days faster than those on placebo within a 30-day window. Notably, this benefit was not apparent at the 25th percentile of recovery time but became highly significant between the 30th and 60th percentiles.

Conclusions and Lessons Learned

The statistical challenges faced by these three trials offer vital lessons for future pandemic research. To ensure the integrity of clinical findings under pressure, the authors recommend:

  1. Prioritizing RMST/RMTR as primary metrics when recovery curves are likely to cross or violate proportional hazard assumptions.
  2. Treating death as a terminal event rather than simple censoring to avoid biasing recovery data.
  3. Maintaining strict control over Type I error if sample sizes must be adjusted mid-trial.
  4. Utilizing Intent-to-Treat (ITT) analysis to preserve the benefits of randomization.

By adopting these more robust statistical frameworks, researchers can provide clearer, more reliable evidence for clinical decision-making during public health emergencies.

Summary updated

Abstract

Abstract Background Since the outbreak of the novel coronavirus disease 2019 (COVID-19) in December 2019, it has rapidly spread in more than 200 countries or territories with over 8 million confirmed cases and 440,000 deaths by June 17, 2020. Recently, three randomized clinical trials on COVID-19 treatments were completed, one for lopinavir-ritonavir and two for remdesivir. One trial reported that remdesivir was superior to placebo in shortening the time to recovery, while the other two showed no benefit of the treatment under investigation. However, several statistical issues in the original design and analysis of the three trials are identified, which might shed doubts on their findings and the conclusions should be evaluated with cautions. Objective From statistical perspectives, we identify several issues in the design and analysis of three COVID-19 trials and reanalyze the data from the cumulative incidence curves in the three trials using more appropriate statistical methods. Methods The lopinavir-ritonavir trial enrolled 39 additional patients due to insignificant results after the sample size reached the planned number, which led to inflation of the type I error rate. The remdesivir trial of Wang et al. failed to reach the planned sample size due to a lack of eligible patients, while the bootstrap method was used to predict the quantity of clinical interest conditionally and unconditionally if the trial had continued to reach the originally planned sample size. Moreover, we used a terminal (or cure) rate model and a model-free metric known as the restricted mean survival time or the restricted mean time to improvement (RMTI) in this context to analyze the reconstructed data due to the existence of death as competing risk and a terminal event. The remdesivir trial of Beigel et al. reported the median recovery time of the remdesivir and placebo groups and the rate ratio for recovery, while both quantities depend on a particular time point representing local information. We reanalyzed the data to report other percentiles of the time to recovery and adopted the bootstrap method and permutation test to construct the confidence intervals as well as the P values. The restricted mean time to recovery (RMTR) was also computed as a global and robust measure for efficacy. Results For the lopinavir-ritonavir trial, with the increase of sample size from 160 to 199, the type I error rate was inflated from 0.05 to 0.071. The difference of terminal rates was −8.74% (95% CI [-21.04, 3.55]; P =.16) and the hazards ratio (HR) adjusted for terminal rates was 1.05 (95% CI [0.78, 1.42]; P =.74), indicating no significant difference. The difference of RMTIs between the two groups evaluated at day 28 was −1.67 days (95% CI [-3.62, 0.28]; P =.09) in favor of lopinavir-ritonavir but not statistically significant. For the remdesivir trial of Wang et al., the difference of terminal rates was −0.89% (95% CI [-2.84, 1.06]; P =.19) and the HR adjusted for terminal rates was 0.92 (95% CI [0.63, 1.35]; P =.67). The difference of RMTIs at day 28 was −0.89 day (95% CI [-2.84, 1.06]; P =.37). The planned sample size was 453, yet only 236 patients were enrolled. The conditional prediction shows that the HR estimates would reach statistical significance if the target sample size had been maintained, and both conditional and unconditional prediction delivered significant HR results if the trial had continued to double the target sample size. For the remdesivir trial of Beigel et al., the difference of RMTRs between the remdesivir and placebo groups up to day 30 was −2.7 days (95% CI [-4.0, −1.2]; P &lt;.001), confirming the superiority of remdesivir. The difference in recovery time at the 25th percentile (95% CI [-3, 0]; P =.65) was insignificant, while the differences manifested to be statistically significant at larger percentiles. Conclusions Based on the statistical issues and lessons learned from the recent three clinical trials on COVID-19 treatments, we suggest more appropriate approaches for the design and analysis for ongoing and future COVID-19 trials.
BibTeX
@misc{yin2020statistical,
  author = {Guosheng Yin and Chenyang Zhang and Huaqing Jin},
  title = {{Statistical Issues and Lessons Learned from COVID-19 Clinical Trials with Lopinavir-Ritonavir and Remdesivir}},
  year = {2020},
  doi = {10.1101/2020.06.17.20133702},
  howpublished = {medRxiv},
}
Figure