Reweighting the RCT for generalization: finite sample error and variable selection

Bénédicte Colnet, Julie Josse, Gaël Varoquaux, Erwan ScornetarXiv 2022 · DOI · arXiv

This paper, titled "Reweighting the RCT for generalization: finite sample error and variable selection," provides a rigorous mathematical framework for a common problem in medicine and policy: how to take the results of a controlled trial and apply them to the messy, diverse real world.

TL;DR

The authors provide the first exact finite-sample analysis of Inverse Propensity of Sampling Weighting (IPSW), a method used to generalize Randomized Controlled Trial (RCT) results to a target population. They prove that estimating treatment probabilities—even when they are known—improves precision and offer a "goldilocks" rule for variable selection: include treatment effect modifiers, but exclude variables that only differ between populations without affecting the treatment's impact.

What Problem Does This Paper Address?

Randomized Controlled Trials (RCTs) are the "gold standard" for determining if a treatment works. However, they often suffer from poor external validity. Trial participants are frequently younger, healthier, or more homogenous than the general population (the "target population"). If a treatment works differently across these groups, the trial's average treatment effect (ATE) will be biased when applied to the real world.

Researchers use IPSW to fix this by reweighting trial participants so their characteristics match the target population. While the theory for very large samples (asymptotics) existed, we lacked a clear understanding of how these estimators behave with finite, real-world sample sizes. Specifically, practitioners didn't know which variables to include in the weights or how the size of the target population sample ($m$) relative to the trial size ($n$) affected the error.

Methods

The authors focus on the Inverse Propensity of Sampling Weighting (IPSW) estimator within the potential outcomes framework. They restrict their analysis to categorical covariates, which allows them to derive exact expressions for bias and variance rather than relying on approximations.

Key methodological contributions include:

  • Finite-Sample Derivations: They establish exact formulas for bias, variance, and quadratic risk for three versions of IPSW: "Oracle" (true probabilities known), "Semi-oracle" (target population known, trial distribution estimated), and "Estimated" (everything learned from data).
  • Regime Analysis: They analyze how the ratio of the trial size ($n$) to the target sample size ($m$) dictates the "variance floor" of the estimate.
  • Variable Selection Theory: They use these formulas to prove how adding different types of covariates (shifted vs. non-shifted, effect modifiers vs. baseline-only) impacts the statistical power of the analysis.

Datasets

The paper validates its theoretical findings using two approaches:

  1. A Toy Simulation: A didactic example involving a genetic mutation that modifies treatment response, used to visualize asymptotic regimes and variance inflation.
  2. Semi-synthetic Medical Data: The authors use real-world covariate distributions from two major sources:
  3. CRASH-3: A large-scale RCT testing Tranexamic Acid for traumatic brain injury.
  4. Traumabase: A high-quality French observational database (the "target population").
  5. Note: They simulate the outcomes ($Y$) on top of these real distributions to ensure they have a "ground truth" to measure against.

Key Insights and Ideas

  • The "Estimated is Better" Paradox: Counter-intuitively, the authors show that estimating the treatment assignment probability ($\pi$) from the trial data leads to lower variance than using the true, "oracle" probability (e.g., 0.5). This is because the estimate adjusts for the specific, random imbalances present in that particular sample.
  • The Variance Stagnation Point: Precision doesn't just depend on the trial size ($n$). If your representative sample of the target population ($m$) is small, increasing the trial size further will eventually yield diminishing returns. The variance is dominated by the smaller of the two datasets.
  • The Danger of "Shifted" Noise: Practitioners often want to include every variable that differs between the trial and the target population. The authors prove this is a mistake. Including a variable that is "shifted" (distributed differently) but is not a treatment effect modifier increases variance without reducing bias.
  • The Benefit of Non-Shifted Modifiers: Conversely, including variables that affect the treatment's impact—even if they are distributed identically in both groups—actually improves the precision of the generalized estimate.

Key Results

  • Consistency Proof: The authors provide a formal proof of $L_2$ consistency for IPSW variants, showing the risk tends to zero as sample sizes increase.
  • Variance Inflation Formula: They provide a concrete formula for how much the variance is multiplied when "bad" covariates are added (Corollary 4).
  • Asymptotic Regimes: They identify that if $m \gg n$, the generalized estimate is essentially as precise as the trial itself. If $m \ll n$, the generalized estimate becomes significantly noisier, dominated by the uncertainty of the target population's demographics.

Limitations and Future Work

  • Categorical Constraint: The primary limitation is the focus on categorical covariates. While continuous variables can be "binned," this can lead to "residual confounding" if the bins are too coarse, or a "curse of dimensionality" if they are too fine.
  • High Dimensionality: The authors acknowledge that as the number of categories grows, the probability of having "empty cells" in the trial increases, which can lead to significant finite-sample bias.
  • Future Directions: The authors suggest extending this finite-sample analysis to continuous covariates using density ratio estimation or more complex "doubly robust" estimators (AIPSW).

Summary updated

Abstract

Randomized Controlled Trials (RCTs) may suffer from limited scope. In particular, samples may be unrepresentative: some RCTs over- or under- sample individuals with certain characteristics compared to the target population, for which one wants conclusions on treatment effectiveness. Re-weighting trial individuals to match the target population can improve the treatment effect estimation. In this work, we establish the exact expressions of the bias and variance of such reweighting procedures -- also called Inverse Propensity of Sampling Weighting (IPSW) -- in presence of categorical covariates for any sample size. Such results allow us to compare the theoretical performance of different versions of IPSW estimates. Besides, our results show how the performance (bias, variance, and quadratic risk) of IPSW estimates depends on the two sample sizes (RCT and target population). A by-product of our work is the proof of consistency of IPSW estimates. Results also reveal that IPSW performances are improved when the trial probability to be treated is estimated (rather than using its oracle counterpart). In addition, we study choice of variables: how including covariates that are not necessary for identifiability of the causal effect may impact the asymptotic variance. Including covariates that are shifted between the two samples but not treatment effect modifiers increases the variance while non-shifted but treatment effect modifiers do not. We illustrate all the takeaways in a didactic example, and on a semi-synthetic simulation inspired from critical care medicine.
BibTeX
@misc{colnet2022reweighting,
  author = {Bénédicte Colnet and Julie Josse and Gaël Varoquaux and Erwan Scornet},
  title = {{Reweighting the RCT for generalization: finite sample error and variable selection}},
  year = {2022},
  eprint = {2208.07614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.07614},
}
Figure