On the two-step hybrid design for augmenting randomized trials using real-world data

Jiapeng Xu, Ruben P. A. van Eijk, Alicia M. Ellis, Tianyu Pan, Lorene M. Nelson, Kit C. B. Roes, Marc van Dijk, Maria Rosaria Sarno, Leonard H. van den Berg, Lü Tian, Ying Lü

TL;DR

Augmenting randomized controlled trials (RCTs) with real-world data (RWD) can significantly speed up drug development for rare diseases, but it risks "borrowing" biased data that leads to false positives. This paper introduces four new statistical calibration methods for a "test-then-pool" design that successfully control these false positive rates (Type I error) while maintaining the efficiency gains of using external data.

What Problem Does This Paper Address?

In traditional clinical trials, patients are split between a treatment group and a control group. For rare diseases like Amyotrophic Lateral Sclerosis (ALS), finding enough participants for a robust control group is a massive hurdle. Researchers want to "borrow" data from existing sources—like disease registries or electronic health records—to bolster the control arm.

However, there is a catch: Exchangeability. If the patients in the real-world registry are fundamentally different from those in the trial (due to different standard-of-care, environment, or hidden biases), pooling them together introduces bias.

Previous researchers proposed a "two-step" approach: first, run a statistical test to see if the RWD and the trial control are equivalent; if they are, pool them. The problem? This "test-then-pool" strategy actually inflates the Type I error rate (the probability of concluding a drug works when it doesn't). Essentially, the act of choosing whether to pool data based on the data itself creates a "p-hacking" style bias that makes the final results less reliable.

Methods

The authors propose four frequentist "calibration" methods to fix the Type I error inflation inherent in the two-step design:

  1. Approach 1 (Large Sample Approximation): It estimates the actual variance of the combined test statistic, accounting for the fact that the distribution is no longer a simple "normal" curve after the first test.
  2. Approach 2 (Exact Threshold): Uses numerical integration to find a specific "critical value" (the bar the results must clear) that keeps the overall error rate at exactly 5%.
  3. Approach 3 (Error Splitting): Pre-allocates a portion of the allowed error rate to the "equivalence" outcome and the rest to the "non-equivalence" outcome.
  4. Approach 4 (Adjusted Borrowing): Only adjusts the "bar for success" when the equivalence test passes and data is borrowed, specifically to cancel out the inflation.

These are compared against the standard "Yuan et al. (2019)" method and the popular Bayesian Power Prior approach, which uses a "discounting" factor to decide how much to trust external data.

Datasets

The researchers evaluated their methods using a synthetic ALS dataset.

  • Source: The data was modeled to match the distribution properties of the Netherlands ALS registry.
  • Context: They simulated a 1:1 randomized, placebo-controlled trial with 200 participants, augmented by 200 matched external controls from the synthetic registry.
  • Metrics: The primary outcome was survival time (time to death), analyzed using Cox proportional hazards models.

Key Insights and Ideas

  • The "Double-Testing" Trap: The paper mathematically proves (Theorem 1) that the original two-step hybrid design always inflates Type I error because it doesn't account for the correlation between the equivalence test and the final treatment effect test.
  • Conservative is Better: Approach 1 (Large Sample Approximation) emerged as the most "conservative" choice. It provided an upper bound on error inflation even when the real-world data was slightly different from the trial data.
  • Frequentist vs. Bayesian: While Bayesian "dynamic borrowing" is popular, the authors found it can lead to massive Type I error inflation (up to 16%) if the real-world data is biased. Their frequentist approaches were much more robust.
  • No Free Lunch: If the real-world data is significantly different from the trial data (non-exchangeable), all methods struggle to control error perfectly. The goal is to minimize that damage.

Key Results

  • Error Control: Under "perfect" conditions (where RWD and trial data are identical), the new methods kept Type I error near the targeted 5%. In contrast, the previous Yuan et al. method hit 6.58%, and the Bayesian Power Prior hit 5.22%.
  • Worst-Case Scenarios: When the RWD was biased, the new methods limited error inflation to 6%–8%. For comparison, the Yuan et al. method reached 10%, and the Bayesian Power Prior spiked to a dangerous 16.78%.
  • Power Gains: By borrowing RWD, the researchers achieved a statistical power of roughly 86%, compared to only 80% if they had used the trial data alone. This means they were more likely to detect a truly effective drug without needing to recruit more patients.

Limitations and Future Work

  • Sample Size Sensitivity: The "test-then-pool" approach requires a large enough control group in the RCT to make the initial equivalence test meaningful. This is still difficult in ultra-rare diseases.
  • Two-Stage Designs: The authors suggest that future work could combine these calibrations with "two-stage" trial designs, where the randomization ratio (how many people get the drug vs. placebo) changes mid-trial based on whether the real-world data looks like a good match.
  • Sensitivity Analysis: They emphasize that in real-world applications, researchers should use multiple methods (sensitivity analysis) rather than relying on a single "best" approach to ensure their findings are robust.

Summary updated

Abstract

Hybrid clinical trials, which borrow real-world data (RWD) from patient registries, claims databases, or electronic health records (EHRs) to augment randomized clinical trials, are of increasing interest. Hybrid clinical trials are especially relevant for rare diseases, where the recruitment of large sample sizes may be challenging. While these trials may better use available information, they assume that the RWD and randomized control arm are exchangeable. Violating this assumption can induce bias, inflate Type I error, or adversely affect statistical power. A two-step hybrid design first tests the exchangeability between randomized control arm and external data sources before incorporating RWD as a comparator for statistical inferences (Yuan et al. 2019). This approach reduces the chance of inappropriate borrowing but may simultaneously inflate the Type I error rate. We propose four different methods to control the Type I error rate under the exchangeability assumption. Approach 1 estimates the variance of the overall test statistic and rejects the null hypothesis based on a Z-test. Approach 2 uses a numerical method to determine the exact critical value for Type I error control. Approach 3 splits the Type I error rates according to the equivalence test outcome. Approach 4 adjusts the critical value only when equivalence is established. We illustrate these methods using a hypothetical scenario in the context of amyotrophic lateral sclerosis (ALS). We evaluate the Type I error and power under various clinical trial conditions in comparison with the Bayesian power prior approach (Ibrahim et al. 2015). We demonstrate that our proposed methods and Bayesian power prior control Type I error and increase power under the exchangeability assumption, whereas the method proposed by Yuan et al. (2019) results in an increased Type I error. In the scenario where the exchangeability assumption does not hold, all methods fail to control the Type I error. Our proposed methods, however, limit a maximum Type I error inflation ranging from 6% to 8%, which compares favorably to 10% for Yuan et al. (2019) and 16% for the Bayesian power prior. All methods increase statistical power under the exchangeability condition but may lead to a loss of statistical power when the exchangeability assumption is violated.
BibTeX
@inproceedings{xu2025two,
  author = {Jiapeng Xu and Ruben P. A. van Eijk and Alicia M. Ellis and Tianyu Pan and Lorene M. Nelson and Kit C. B. Roes and Marc van Dijk and Maria Rosaria Sarno and Leonard H. van den Berg and Lü Tian and Ying Lü},
  title = {{On the two-step hybrid design for augmenting randomized trials using real-world data}},
  booktitle = {Statistics},
  year = {2025},
  doi = {10.1080/19466315.2025.2547855},
}

Loading related content...

Figure