Determining sample size for pilot trials: a tutorial

Xiangji Ying, Kenneth E. Freedland, Lynda H. Powell, Elizabeth A. Stuart, Stephan Ehrhardt, Evan Mayo‐WilsonBMJ 2025 · DOI

Abstract

Pilot trials have a key role in preparing for definitive randomized trials, yet determining their sample size remains a common challenge. This article provides practical guidance, methods, and tools—including step-by-step examples, sample size tables, and statistical code—for calculating and justifying sample sizes in external randomized pilot trials.

Introduction

In clinical research, definitive randomized controlled trials (RCTs) represent the gold standard for evaluating interventions. However, the path to a successful RCT is fraught with logistical risks; approximately 9% of these trials are terminated prematurely due to recruitment failures or operational glitches. Pilot trials serve as essential "dress rehearsals" to mitigate these risks, yet they are frequently sized using arbitrary "rules of thumb" or the persistent "30 per group" myth.

The fundamental challenge lies in the fact that traditional power calculations—the standard mathematical approach for sizing main trials—are inappropriate for pilots. Pilot trials are not intended to test efficacy, but rather to assess feasibility. This tutorial addresses this gap by providing a rigorous framework to match sample size calculations to specific pilot objectives.

Methodological Frameworks

The authors move away from a "one-size-fits-all" approach, categorizing pilot trial objectives into five distinct methodological frameworks. These frameworks allow researchers to justify sample sizes based on the specific feasibility questions they need to answer.

Table 1: Overview of methods to determine pilot trial sample size.
  1. Precision Analysis: This is used when the primary goal is to estimate a specific parameter, such as a dropout rate, within a defined margin of error.
  2. One-Sample Power Analysis: This framework supports "Traffic Light" progression criteria. By setting thresholds for success (Green), modification (Amber), or failure (Red), researchers can use power analysis to determine the sample size needed to make a reliable "go/no-go" decision.
  3. Detection Probability: This approach ensures a high level of confidence (e.g., 95%) that at least one occurrence of a specific logistical "bug" or adverse event will be observed during the pilot.
  4. Variability Estimation & Inflation: Pilots are often used to estimate the standard deviation of a primary outcome. Because small-sample estimates are uncertain, this method involves "inflating" the subsequent main trial's size to ensure it remains adequately powered.
  5. Total Sample Size Optimization: This iterative process identifies the "sweet spot" pilot size that minimizes the total number of participants required across both the pilot and the definitive trial.

Practical Tools and Resources

To facilitate the application of these methods, the tutorial is supported by a suite of open-source tools. This includes the PilotSize R Shiny application and comprehensive statistical code for R, Stata, and SAS. Furthermore, the authors provide pre-calculated sample size tables derived from relevant statistical distributions, such as Poisson and Binomial, aligning with CONSORT extension guidelines for pilot and feasibility studies.

Key Insights

The Efficacy Trap

A critical warning provided by the authors is the danger of using pilot trials to estimate "preliminary efficacy." Due to their small size, pilots produce imprecise effect size estimates that are highly susceptible to publication bias. Relying on these estimates to size a main trial frequently leads to underpowered studies and "research waste."

The U-Shaped Curve

There is a mathematical trade-off between the size of a pilot and the size of the subsequent main trial. While a larger pilot requires more initial resources, it reduces the "inflation factor" needed for the main trial by providing a more precise variability estimate. The authors demonstrate that there is a specific point on this U-shaped curve where the combined sample size of both trials is minimized.

Strategic Decision Making

The "Traffic Light" system and "Rule Out" strategies offer formal ways to handle uncertainty. For instance, researchers can use a pilot to "rule out" interventions that show zero or negative results using a one-sided 80% confidence interval, thereby saving significant resources that would otherwise be spent on a futile definitive trial.

Based on the frameworks provided, the paper offers concrete benchmarks for common scenarios:

  • Problem Detection: To detect a problem with a 10% probability at 95% confidence, a sample of 29-30 participants is required.
  • Variability Estimation: To balance precision and efficiency when estimating variability for a main trial, a pilot of 12 to 35 participants per group is recommended.
  • Recruitment Testing: To test recruitment uptake (e.g., distinguishing a 20% minimum from a 50% goal), a sample of 28 participants (14 per group) provides 95% power for the decision.

Limitations and Future Directions

While this tutorial provides a robust foundation, it focuses primarily on external, fixed, two-arm, individually randomized trials. It does not address more complex designs such as cluster randomized trials or adaptive designs. Additionally, the authors note that pilot participants may not always perfectly represent the eventual definitive trial population, potentially leading to "optimism bias." Finally, while the paper focuses on frequentist statistics, it highlights Bayesian approaches—which incorporate prior knowledge—as a promising area for future small-sample pilot decision-making.

Summary updated

Abstract

Pilot trials have a key role in preparing for definitive randomized trials, yet determining their sample size remains a common challenge. This article provides practical guidance, methods, and tools—including step-by-step examples, sample size tables, and statistical code—for calculating and justifying sample sizes in external randomized pilot trials.
BibTeX
@inproceedings{ying2025determining,
  author = {Xiangji Ying and Kenneth E. Freedland and Lynda H. Powell and Elizabeth A. Stuart and Stephan Ehrhardt and Evan Mayo‐Wilson},
  title = {{Determining sample size for pilot trials: a tutorial}},
  booktitle = {BMJ},
  year = {2025},
  doi = {10.1136/bmj-2024-083405},
}
Figure