Multivariate Dynamic Mediation Analysis under a Reinforcement Learning Framework

Lan Luo, Chengchun Shi, Jitao Wang, Zhenke Wu, Lexin LiThe Annals of Statistics 2023 · arXiv

Abstract

Mediation analysis is an important analytic tool commonly used in a broad range of scientific applications. In this article, we study the problem of mediation analysis when there are multivariate and conditionally dependent mediators, and when the variables are observed over multiple time points. The problem is challenging, because the effect of a mediator involves not only the path from the treatment to this mediator itself at the current time point, but also all possible paths pointed to this mediator from its upstream mediators, as well as the carryover effects from all previous time points. We propose a novel multivariate dynamic mediation analysis approach. Drawing inspiration from the Markov decision process model that is frequently employed in reinforcement learning, we introduce a Markov mediation process paired with a system of time-varying linear structural equation models to formulate the problem. We then formally define the individual mediation effect, built upon the idea of simultaneous interventions and intervention calculus. We next derive the closed-form expression and propose an iterative estimation procedure under the Markov mediation process model. We study both the asymptotic property and the empirical performance of the proposed estimator, and further illustrate our method with a mobile health application.

Introduction

In classical mediation analysis, researchers seek to understand how a treatment ($A$) influences an outcome ($R$) through an intermediate variable ($M$). However, real-world systems, particularly in fields like mobile health (mHealth), are rarely so simple. A single intervention, such as a smartphone notification, might simultaneously affect physical activity, sleep, and heart rate, which then collectively influence a long-term outcome like mood.

Existing methodologies often fail to address three critical hurdles simultaneously:

  1. Multivariate Dependency: Mediators are frequently dependent on one another.
  2. Temporal Dynamics: Variables observed over time create "carryover effects," where past states influence future outcomes.
  3. Path Complexity: As the number of mediators and time points increases, the number of potential causal paths grows super-exponentially, rendering traditional path-summing methods computationally intractable.

The Markov Mediation Process

To overcome these challenges, this research introduces the Markov Mediation Process (MMP). This framework draws direct inspiration from the Markov Decision Processes (MDPs) used in Reinforcement Learning (RL). By viewing mediation as a state-transition problem, the authors leverage the "memoryless" property of Markov processes to simplify complex causal webs.

Comparison between a standard Markov Decision Process (a) and the proposed Markov Mediation Process (b).

The system is modeled using a series of time-varying linear Structural Equation Models (SEMs). This allows for the characterization of how the current state of mediators depends on both the treatment and previous states. Using Judea Pearl’s intervention calculus (the "do-operator"), the authors formally define the Individual Mediation Effect (IME). A significant innovation of this approach is the decomposition of the IME into two distinct components:

  • Immediate Effect: The change in the outcome at the current time point.
  • Delayed Effect: The "carryover" impact on future outcomes.

To solve the computational challenges of path complexity, the authors derived closed-form expressions and an efficient recursive algorithm, enabling the calculation of effects across many time steps without manually tracing every possible path.

Empirical Evaluation and Results

The proposed MMP method was validated using both synthetic simulations and real-world data from the Intern Health Study (IHS). The IHS followed 1,196 first-year physicians over 26 weeks, investigating how mobile health prompts mediated mood through physiological markers like step count, sleep duration, and resting heart rate (RHR).

Key findings include:

  • Statistical Superiority: In simulations, the MMP method showed lower bias and better coverage probabilities than baseline methods that ignored temporal dynamics or mediator correlations.
  • Mobile Health Insights: The study revealed that push notifications often led to increased RHR and reduced sleep hours, both of which were associated with lower mood scores. Crucially, the delayed effects were often more pronounced than immediate ones, proving that the impact of health interventions often "brews" over time.

Conclusion and Future Directions

The MMP framework offers a powerful new tool for disentangling complex, time-varying causal relationships. While the current model assumes linearity and the absence of unobserved confounders, it provides a robust foundation for future work. Extending this framework to non-linear models and developing methods for observational data remain important next steps in the evolution of dynamic mediation analysis.

Summary updated

Abstract

Mediation analysis is an important analytic tool commonly used in a broad range of scientific applications. In this article, we study the problem of mediation analysis when there are multivariate and conditionally dependent mediators, and when the variables are observed over multiple time points. The problem is challenging, because the effect of a mediator involves not only the path from the treatment to this mediator itself at the current time point, but also all possible paths pointed to this mediator from its upstream mediators, as well as the carryover effects from all previous time points. We propose a novel multivariate dynamic mediation analysis approach. Drawing inspiration from the Markov decision process model that is frequently employed in reinforcement learning, we introduce a Markov mediation process paired with a system of time-varying linear structural equation models to formulate the problem. We then formally define the individual mediation effect, built upon the idea of simultaneous interventions and intervention calculus. We next derive the closed-form expression and propose an iterative estimation procedure under the Markov mediation process model. We study both the asymptotic property and the empirical performance of the proposed estimator, and further illustrate our method with a mobile health application.
BibTeX
@article{luo2023multivariate,
  author = {Lan Luo and Chengchun Shi and Jitao Wang and Zhenke Wu and Lexin Li},
  title = {{Multivariate Dynamic Mediation Analysis under a Reinforcement Learning Framework}},
  journal = {The Annals of Statistics},
  year = {2023},
}
Figure