Selection bias in the treatment effect for a principal stratum

Yongming Qu, Stephen J. Ruberg, Junxiang Luo, Ilya LipkovicharXiv 2021 · DOI · arXiv

Abstract

Estimation of treatment effect for principal strata has been studied for more than two decades. Existing research exclusively focuses on the estimation, but there is little research on forming and testing hypotheses for principal stratification-based estimands. In this brief report, we discuss a phenomenon in which the true treatment effect for a principal stratum may not equal zero even if the two treatments have the same effect at patient level which implies an equal average treatment effect for the principal stratum. We explain this phenomenon from the perspective of selection bias. This is an important finding and deserves attention when using and interpreting results based on principal stratification. There is a need to further study how to form the null hypothesis for estimands for a principal stratum.

Introduction

In clinical trials, researchers frequently encounter "intercurrent events"—circumstances such as treatment discontinuation due to adverse effects or lack of efficacy—that complicate the analysis of a drug's true impact. Principal stratification has emerged as a robust framework to address these events by categorizing patients into strata based on how they would behave under different treatment assignments. For example, a common stratum of interest is the group of patients who would adhere to the experimental treatment regardless of their assignment.

However, a fundamental assumption has long underpinned this field: if a treatment has zero effect on every individual in a population ( the "null scenario"), then the average treatment effect within any given principal stratum must also be zero. This paper demonstrates that this assumption is often mathematically incorrect due to inherent selection bias, posing a significant challenge to standard null hypothesis testing.

The Mechanism of Selection Bias

The core issue lies in how strata are defined. The authors contrast two primary types of principal strata:

  1. Always-Adherers ($S_{++}$): Patients who would adhere to both the experimental treatment and the control.
  2. Experimental-Adherers ($S_{*+}$): Patients who would adhere to the experimental treatment, regardless of whether they would adhere to the control.

When a stratum is defined by behavior under only one treatment arm (as in $S_{*+}$), it introduces an asymmetric selection bias. If the factors that influence a patient's likelihood to adhere to a drug—such as baseline health status or early physiological responses—are also correlated with the final clinical outcome, the act of selecting that stratum effectively "hand-picks" a group with specific outcome profiles. Consequently, even if the drug itself is completely ineffective, the stratum may show a non-zero treatment effect simply because of the characteristics of the people within it.

Methodology and Mathematical Proof

To prove the existence of this bias, the authors utilized a potential outcomes framework and a theoretical data generation model established in previous literature (Qu et al., 2020). This simulation framework allowed for a controlled environment where the treatment effect was known to be zero at the individual level.

In the model:

  • Adherence was determined via logistic regression using baseline covariates and intermediate measurements.
  • Outcomes were modeled as linear functions of those same variables.

By deriving the expected treatment effect for these groups, the researchers proved that while the effect for the symmetric "always-adherers" ($S_{++}$) remains zero under the null, the effect for the "experimental-adherers" ($S_{*+}$) results in a complex integral that does not resolve to zero. This confirms that $\mu_{d,*+} \neq 0$ even when the drug has no biological effect.

Key Insights

  • Symmetry as a Safeguard: Strata defined by behavior under both treatment arms (like $S_{++}$) are naturally protected from selection bias because the selection criteria are applied equally to both potential outcomes, maintaining a balance between the groups.
  • The Fallacy of the Zero Null: In principal stratification, the "true" treatment effect under a null hypothesis is not always zero. This is counter-intuitive to standard statistical practice and suggests that current methods for testing hypotheses in these models may be prone to false positives.
  • Conditions for Unbiasedness: The authors identified a specific "sufficient condition"—$\{Y(1) - Y(0)\} \perp A(1)|X$—under which the bias disappears. This requires the treatment effect to be independent of adherence, given the baseline covariates. However, the authors note that this condition is highly restrictive and unlikely to hold in most real-world clinical settings.

Conclusion and Future Directions

This research serves as a critical warning for statisticians and clinicians using principal stratification. The discovery that selection bias can create a phantom treatment effect necessitates a re-evaluation of how null hypotheses are formed and tested.

Future work is required to develop new methodologies that account for this bias, such as "random split" techniques to estimate natural selection effects. Additionally, further investigation is needed into how this phenomenon impacts non-inferiority studies, where the mathematical implications of selection bias may be even more consequential than in superiority trials.

Summary updated

Abstract

Estimation of treatment effect for principal strata has been studied for more than two decades. Existing research exclusively focuses on the estimation, but there is little research on forming and testing hypotheses for principal stratification-based estimands. In this brief report, we discuss a phenomenon in which the true treatment effect for a principal stratum may not equal zero even if the two treatments have the same effect at patient level which implies an equal average treatment effect for the principal stratum. We explain this phenomenon from the perspective of selection bias. This is an important finding and deserves attention when using and interpreting results based on principal stratification. There is a need to further study how to form the null hypothesis for estimands for a principal stratum.
BibTeX
@misc{qu2021selection,
  author = {Yongming Qu and Stephen J. Ruberg and Junxiang Luo and Ilya Lipkovich},
  title = {{Selection bias in the treatment effect for a principal stratum}},
  year = {2021},
  eprint = {2112.09541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.09541},
}
Figure