How to estimate the association between change in a risk factor and a health outcome?

Michail Katsoulis, Alvina G. Lai, Dimitra‐Kleio Kipourou, Reecha Sofat, Manuel Gomes, Amitava Banerjee, Spiros Denaxas, Thomas R Lumbers, Konstantinos K. Tsilidis, Harry Hemingway, Karla Díaz-OrdazarXiv 2020 · DOI · arXiv

Abstract: Estimating the effect of a change in a particular risk factor and a chronic disease requires information on the risk factor from two time points; the enrolment and the first follow-up. When using observational data to study the effect of such an exposure (change in risk factor) extra complications arise, namely (i) when is time zero? and (ii) which information on confounders should we account for in this type of analysis? From enrolment or the 1st follow-up? Or from both?. The combination of these questions has proven to be very challenging. Researchers have applied different methodologies with mixed success, because the different choices made when answering these questions induce systematic bias. Here we review these methodologies and highlight the sources of bias in each type of analysis. We discuss the advantages and the limitations of each method ending by making our recommendations on the analysis plan.

Introduction

A fundamental challenge in epidemiology is determining whether changing a health behavior—such as losing weight or lowering blood pressure—actually improves long-term health outcomes. While Randomized Controlled Trials (RCTs) are the gold standard for answering these questions, they are often prohibitively expensive or impractical for long-term chronic disease monitoring. Consequently, researchers frequently turn to observational data, such as Electronic Health Records (EHR).

However, observational data lacks the controlled environment of a trial. It records measurements but not intentions. This creates a significant risk of bias, particularly regarding reverse causality. For instance, if a patient’s BMI drops significantly, was it due to a healthy lifestyle change or an undiagnosed underlying illness? This paper by Katsoulis et al. evaluates how different analytical frameworks handle these complexities and identifies the most robust method for estimating the impact of risk factor changes.

The Problem: Time Zero and Confounding

The authors identify two core questions that often lead to confusion and bias in observational studies:

  1. When is "Time Zero"? Should the follow-up for the health outcome begin at the initial measurement (enrollment) or after the change in the risk factor has been completed (the first follow-up)?
  2. Which confounders matter? Should researchers adjust for health status at the start of the study, at the end of the change period, or both?

Failure to answer these questions correctly often results in capturing "intermediate events." If a patient becomes ill during the period their weight is being monitored, that illness might cause weight loss and simultaneously increase the risk of the primary outcome (e.g., heart disease). Without proper adjustment, a researcher might incorrectly conclude that weight loss causes heart disease.

Methodological Framework

The authors utilize a Target Trial Emulation framework to compare four common methodologies:

  • Method 1: Time zero is set at enrollment. It adjusts only for confounders present at the start of the study.
  • Method 2: Time zero is set at the first follow-up (after the change period). It adjusts only for confounders measured at that follow-up.
  • Method 3: Time zero is at the first follow-up, but the analysis excludes any individuals who developed the outcome or significant illness during the "change period."
  • Method 4 (The Recommended Approach): This method defines a "baseline period" (the window between enrollment and first follow-up). Time zero is technically at enrollment, but the analysis adjusts for confounders measured at both time points. Crucially, it excludes individuals who develop chronic diseases during this window to isolate the effect of the risk factor change from pre-existing illness.

Validation via Simulation

To evaluate these methods, the researchers used a hypothetical dataset of 80 individuals representing a larger population. The advantage of a simulated dataset is that the "true" causal effect is defined by the researchers. By comparing the results of each method against the known truth, they could precisely measure the bias introduced by each approach.

Key Results and Insights

The simulation revealed that common analytical choices can lead to "completely wrong" conclusions:

  • The Failure of Traditional Approaches: Methods 1 and 2 performed poorly. In a scenario where weight loss was actually neutral, these methods estimated a 7.1% risk difference, suggesting weight loss was harmful. This was due to reverse causality—the methods failed to account for weight loss driven by undiagnosed illness.
  • The Success of Method 4: Method 4 successfully recovered the true effect. In the obese cohort, it estimated a beneficial effect within 0.6% of the true "gold standard" trial result.
  • The Power of the Baseline Period: By treating the baseline as a duration rather than a single point, Method 4 "blocks" backdoor pathways (confounders) that appear after the study starts but before the exposure is fully realized.

Limitations and Conclusion

While Method 4 is the most robust, it is not without trade-offs. It requires the assumption that individuals survive and remain healthy until the first follow-up, which introduces a minor "immortal time bias." However, the authors demonstrate that this bias is negligible compared to the massive confounding errors found in other methods.

Furthermore, the authors note the Consistency Assumption: observational data on "BMI change" cannot distinguish between weight lost via exercise versus weight lost via medication or surgery. Despite this, the recommended framework provides a significantly more accurate way to re-evaluate fluctuating risk factors—such as blood pressure or cholesterol—in existing observational datasets. For future research, adopting the "baseline period" approach is essential for minimizing systematic error in longitudinal health studies.

Summary updated

Abstract

Estimating the effect of a change in a particular risk factor and a chronic disease requires information on the risk factor from two time points; the enrolment and the first follow-up. When using observational data to study the effect of such an exposure (change in risk factor) extra complications arise, namely (i) when is time zero? and (ii) which information on confounders should we account for in this type of analysis? From enrolment or the 1st follow-up? Or from both?. The combination of these questions has proven to be very challenging. Researchers have applied different methodologies with mixed success, because the different choices made when answering these questions induce systematic bias. Here we review these methodologies and highlight the sources of bias in each type of analysis. We discuss the advantages and the limitations of each method ending by making our recommendations on the analysis plan.
BibTeX
@misc{katsoulis2020how,
  author = {Michail Katsoulis and Alvina G. Lai and Dimitra‐Kleio Kipourou and Reecha Sofat and Manuel Gomes and Amitava Banerjee and Spiros Denaxas and Thomas R Lumbers and Konstantinos K. Tsilidis and Harry Hemingway and Karla Díaz-Ordaz},
  title = {{How to estimate the association between change in a risk factor and a health outcome?}},
  year = {2020},
  eprint = {2012.11291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.11291},
}
Figure