Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings

Wenxin Chen, Weishen Pan, Kyra Gan, Fei Wang

Introduction: The Flaw of Separate Estimation

In healthcare and public policy, decisions are rarely isolated events; they are longitudinal processes. A clinician administers a treatment, the patient's vitals (time-varying confounders) change in response, and the clinician subsequently adjusts the treatment. This feedback loop makes it notoriously difficult to estimate counterfactuals—that is, predicting what would have happened had a different treatment strategy been followed.

Current state-of-the-art causal inference methods, such as Longitudinal Targeted Maximum Likelihood Estimation (LTMLE), typically evaluate potential policies independently. For example, comparing ten different ventilator weaning strategies requires training ten separate models. The authors identify a critical flaw in this "separate estimation" paradigm: it ignores the structural similarities between policies. Because models are trained in isolation, even nearly identical policies will yield divergent estimates due to random initialization and sampling noise. This induces a structurally uncontrolled second-order bias, leading to high variance and unreliable comparisons, particularly when evaluating closely related policies.

Methodology: The PEQ-Net Architecture

To address this variance, the researchers developed the Policy-Encoded Q Network (PEQ-Net), a novel deep learning architecture that evaluates multiple medical treatment strategies simultaneously. By embedding different policies into a shared mathematical space, the model shares information across similar scenarios.

PEQ-Net relies on two primary innovations:

  1. Policy-Aware Reparameterization: Instead of training isolated models to predict outcomes for specific policies, PEQ-Net redefines the "Q-function" (the expected future outcome) to take the future policy itself as an input. This allows a single neural network to learn outcomes for an entire family of policies simultaneously.
  2. RKHS-Based Policy Embedding: To feed a complex policy function into a neural network, it must be vectorized. The authors utilize Kernel Mean Embeddings in a Reproducing Kernel Hilbert Space (RKHS) to measure the distance between policies based on their population-level behavior. Multidimensional Scaling (MDS) is then used to project these distances into a low-dimensional vector.

The underlying architecture employs a Transformer backbone to process time-series patient data alongside an RNN-based policy encoder for the future treatment strategy. Finally, a standard LTMLE debiasing step is applied to guarantee statistical robustness.

Experimental Setup

The model's efficacy was tested across three distinct scenarios:

  • Limited Time-Varying Confounding: A semi-synthetic dataset utilizing 10 real-world clinical variables from the MIMIC-III database, where treatments affect outcomes but not future patient vitals.
  • Expanded Time-Varying Confounding: A complex synthetic environment where treatments directly impact future patient states, simulating difficult clinical feedback loops.
  • Real-World Case Study: An analysis of 999 sepsis patients from the MIMIC-IV database, estimating the effect of various Mean Arterial Pressure (MAP) targets on serum lactate levels (a key marker of organ failure).

Key Insights and Results

The fundamental breakthrough of PEQ-Net is its mathematical guarantee of smoothness. The authors prove that utilizing a shared representation imposes a "structural constraint" on the estimation error. Because the model maps similar policies to nearby points in vector space, it is physically constrained from producing wildly erratic outcome estimates for them. By ensuring the model is Lipschitz continuous, small changes in policy reliably lead to small changes in predicted outcomes.

Empirically, this translates to massive performance gains:

  • Error Reduction: On semi-synthetic data, PEQ-Net achieved substantial reductions in Root-Mean-Square Error (RMSE) compared to baselines like DeepLTMLE and DeepACE, cutting bias by over 50% in certain scenarios.
  • Stability: Performance improvements were most dramatic when evaluating closely related policies—the exact domain where traditional isolated methods succumb to noise.
  • Clinical Alignment: In the MIMIC-IV sepsis case study, PEQ-Net demonstrated that more aggressive vasopressor use (higher MAP targets) was associated with higher lactate levels. This aligns seamlessly with recent randomized controlled trials, such as the SEPSISPAM trial, which suggest that higher targets do not necessarily improve survival and may introduce additional risks.

Limitations and Future Directions

While PEQ-Net offers a robust solution to longitudinal causal inference, it has a few limitations. The policy embedding step requires calculating a distance matrix between all policies and patient trajectories, resulting in a quadratic computational complexity ($O(N^2)$). While approximation techniques like the Nyström method could mitigate this, scalability remains a hurdle for massive datasets.

Furthermore, the current framework requires users to pre-define the set of policies and focuses primarily on binary treatment assignments. Future iterations could allow the model to autonomously "discover" optimal policy groupings and extend the architecture to handle continuous treatment dosages, broadening its applicability in dynamic clinical environments.

Summary updated

Abstract

Comparative evaluation of *multiple* dynamic treatment policies is essential for healthcare and policy decisions, yet conventional longitudinal causal inference methods estimate each in *isolation*, preventing information sharing across counterfactuals. We demonstrate that this separate estimation paradigm induces a structurally uncontrolled second-order bias, inflating finite-sample variance even after standard debiasing with *longitudinal targeted maximum likelihood estimation* (LTMLE). To address this, we propose a policy-aware reparameterization of *Iterative Conditional Expectation* (ICE) Q-functions that enables joint estimation through shared representations. We implement this approach in the **Policy-Encoded Q Network (PEQ-Net)**, an architecture centered on a shared policy encoder. The encoder is trained using kernel mean embeddings, ensuring that the learned representation space reflects population-level policy dissimilarities. After applying an LTMLE correction step, we prove this design imposes a structural constraint on the second-order remainder, thereby stabilizing finite-sample variance. Experiments on semi-synthetic datasets demonstrate that PEQ-Net consistently outperforms existing ICE-based methods, achieving substantial reductions in root-mean-square error, particularly when evaluating closely related policies.
BibTeX
@inproceedings{chen2026smooth,
  author = {Wenxin Chen and Weishen Pan and Kyra Gan and Fei Wang},
  title = {{Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=bIcz7bIZSo},
}
Figure