Comparison of Bayesian Networks, G-estimation and linear models to estimate causal treatment effects in aggregated N-of-1 trials with carry-over effects

Thomas Gärtner, Juliana Schneider, Bert Arnrich, Stefan KonigorskimedRxiv 2022 · DOI

Abstract

The aggregation of a series of N-of-1 trials presents an innovative and efficient study design, as an alternative to traditional randomized clinical trials. Challenges for the statistical analysis arise when there is carry-over or complex dependencies of the treatment effect of interest. In this study, we evaluate and compare methods for the analysis of aggregated N-of-1 trials in different scenarios with carry-over and complex dependencies of treatment effects on covariates. For this, we simulate data of a series of N-of-1 trials for Chronic Nonspecific Low Back Pain based on assumed causal relationships parameterized by directed acyclic graphs. In addition to existing statistical methods such as regression models, Bayesian Networks, and G-estimation, we introduce a carry-over adjusted parametric model (COAPM). The results show that all evaluated existing models have a good performance when there is no carry-over and no treatment dependence. When there is carry-over, COAPM yields unbiased and more efficient estimates while all other methods show some bias in the estimation. This study presents a systematic evaluation of existing and novel approaches for the statistical analysis of a series of N-of-1 trials.

Introduction

N-of-1 trials—multi-crossover controlled trials where a single patient serves as their own control—are the gold standard for determining individual treatment efficacy. When these trials are aggregated across many patients, they can provide population-level insights with higher efficiency than traditional Randomized Controlled Trials (RCTs). However, two major statistical challenges haunt aggregated N-of-1 data: carry-over effects, where the physiological impact of a previous treatment persists into the next period, and time-varying confounding, where factors like daily activity influence both treatment adherence and health outcomes.

Methods

The researchers compared five distinct statistical approaches to estimate treatment effects:

  • Sample Mean Comparison: A naive baseline comparing average outcomes.
  • Standard Linear Regression: Adjusting for covariates but treating treatment as a simple binary variable.
  • Adjusted Linear Model (COAPM): This novel approach replaces the binary treatment variable with a continuous, time-dependent variable that models exponential "wash-in" and "wash-out" effects.
  • Bayesian Networks (BN): Graphical models that estimate joint probability distributions, tested with and without time-lagged variables.
  • G-estimation: A causal inference method designed to handle time-varying confounding using structural nested mean models.

Simulation Framework

Because the "true" causal effect is unknown in real-world clinical data, the authors developed a Monte Carlo simulation framework based on chronic nonspecific low back pain. The data generation used Directed Acyclic Graphs (DAGs) to define complex relationships between demographics, activity, stress, sleep quality, and pain.

Causal DAG for Chronic Nonspecific Low Back Pain simulation

Four primary scenarios were generated: Baseline (no carry-over or confounding), Carry-over only, Confounding only, and a "Worst Case" combining both. They also simulated missing data to mimic patient dropouts and vacations.

Key Results

The performance of the models varied significantly depending on the presence of carry-over and confounding effects.

Comparison of treatment effect estimates across scenarios and sample sizes
  • Baseline: All methods provided unbiased estimates of the true treatment effect (set at 2.0).
  • Carry-over Effects: This proved to be a "silent killer" of accuracy. In scenarios with carry-over, every standard model (including G-estimation and basic BNs) significantly underestimated the treatment effect. Only the proposed Adjusted Linear Model remained unbiased.
  • Confounding: The sample mean failed significantly (bias > 50%), but all other models that included the confounding variable (Activity) successfully recovered the true effect.
  • Sample Size and Efficiency: Bayesian Networks were robust but yielded much wider confidence intervals for small sample sizes (e.g., $n=5$) compared to linear models, making them less efficient for small pilot studies.

Discussion and Recommendations

Carry-over effects are often ignored in the analysis of N-of-1 trials, yet this study demonstrates they can lead to substantial bias. Simple time lags in Bayesian Networks were insufficient to capture the complex exponential decay of treatment effects.

The authors recommend the use of the Adjusted Linear Model (COAPM) when carry-over is suspected. While G-estimation is powerful for confounding, its performance is highly sensitive to the assumed correlation structure. For researchers looking to implement these methods, the authors have released an R package, cinof1, to facilitate more accurate causal inference in personalized medicine.

Summary updated

Abstract

A bstract The aggregation of a series of N-of-1 trials presents an innovative and efficient study design, as an alternative to traditional randomized clinical trials. Challenges for the statistical analysis arise when there is carry-over or complex dependencies of the treatment effect of interest. In this study, we evaluate and compare methods for the analysis of aggregated N-of-1 trials in different scenarios with carry-over and complex dependencies of treatment effects on covariates. For this, we simulate data of a series of N-of-1 trials for Chronic Nonspecific Low Back Pain based on assumed causal relationships parameterized by directed acyclic graphs. In addition to existing statistical methods such as regression models, Bayesian Networks, and G-estimation, we introduce a carry-over adjusted parametric model (COAPM). The results show that all evaluated existing models have a good performance when there is no carry-over and no treatment dependence. When there is carry-over, COAPM yields unbiased and more efficient estimates while all other methods show some bias in the estimation. When there is known treatment dependence, all approaches that are capable to model it yield unbiased estimates. Finally, the efficiency of all methods decreases slightly when there are missing values, and the bias in the estimates can also increase. This study presents a systematic evaluation of existing and novel approaches for the statistical analysis of a series of N-of-1 trials. We derive practical recommendations which methods may be best in which scenarios.
BibTeX
@misc{gartner2022comparison,
  author = {Thomas Gärtner and Juliana Schneider and Bert Arnrich and Stefan Konigorski},
  title = {{Comparison of Bayesian Networks, G-estimation and linear models to estimate causal treatment effects in aggregated N-of-1 trials with carry-over effects}},
  year = {2022},
  doi = {10.1101/2022.07.21.22277832},
  howpublished = {medRxiv},
}
Figure