Target-Driven Policy Optimization for Sequential Counterfactual Outcome Control

Xin Wang, Xiangyu Zhang, Shengfei Lyu, Huanhuan Chen

The Challenge of Personalized Treatment Planning

In personalized medicine, a primary objective is leveraging Electronic Health Records (EHRs) to determine the optimal sequence of treatments to achieve a specific health goal, such as reducing a tumor to a target size or stabilizing blood glucose levels. This challenge is formally known as Sequential Counterfactual Target Achievement (SCTA).

Historically, state-of-the-art methods have relied on a "learn-then-plan" paradigm. These models first learn to predict counterfactual outcomes and then run a computationally expensive optimization search at the time of treatment to find a fixed sequence of actions. This approach suffers from three major limitations:

  1. Inefficiency: The optimization search is too slow for real-time clinical decision support.
  2. Lack of Adaptability: The resulting plans are "open-loop," meaning they cannot easily adjust if a patient's health trajectory deviates from the initial prediction.
  3. Data Confounding and Sparsity: Observational data is inherently confounded (doctors choose treatments based on patient history), rewards are sparse (patients rarely hit the exact target in historical data), and there is a massive distributional shift between historical practice and the optimal policy.

Introducing GIFT: A Real-Time, Reactive Policy

To overcome these hurdles, researchers propose GIFT (Goal-conditioned Intervention via Factual-Targeted Training). GIFT reformulates SCTA as a Goal-conditioned Markov Decision Process (MDP). Instead of searching for a static plan, GIFT learns a dynamic policy—a function that evaluates the patient's current state and target goal to immediately output the next best action.

Key Technical Innovations

  • Hindsight Experience Replay (HER): To solve the sparse reward problem, GIFT relabels "failed" historical trajectories. If a patient did not reach the intended target but ended up at state X, GIFT treats state X as a "virtual goal." This teaches the model how to reach useful states even from suboptimal historical data.
  • Clipped Reward Rescaling: To handle distributional shifts and confounding variables, GIFT uses importance sampling weights to rescale rewards. By clipping these weights, the framework stabilizes the training process, carefully balancing the bias of the estimator against the high variance typically found in offline Reinforcement Learning (RL).
  • Soft Actor-Critic (SAC) Backbone: The policy is trained using SAC, which is well-suited for the continuous, "dosage-like" actions common in medicine and improves training stability through entropy regularization.

Experimental Validation

The researchers evaluated GIFT on two complex environments:

  1. Tumor Dataset: A pharmacokinetic-pharmacodynamic simulator modeling tumor growth under radiotherapy and chemotherapy. It includes a confounding parameter ($\kappa$) to test the model's robustness against biased historical data.
  2. MIMIC-III (Semi-synthetic): Based on real-world ICU data, this dataset uses actual physiological histories but synthesizes treatment effects to provide a ground truth for evaluation—a necessity when working with purely observational data.

Key Findings

  • Superior Accuracy: Across both datasets, GIFT consistently achieved the lowest Root Mean Square Error (RMSE) compared to the target goal. In highly confounded scenarios (e.g., $\kappa=4$ in the Tumor dataset), GIFT's error was less than half that of the best-performing baseline.
  • Rapid Inference: Because GIFT utilizes a direct policy rollout rather than an iterative optimization search, its inference time is orders of magnitude faster than traditional "learn-then-plan" methods.
  • Robust Generalization: GIFT demonstrated strong performance even when tasked with reaching "unseen" targets that were not well-represented in the original training data.

Theoretical Insights and Future Directions

A major theoretical contribution of this work is bridging causal inference and reinforcement learning. The authors prove that their modified Bellman operator is "causally identifiable," meaning the policy learned from biased observational data is mathematically valid for making real-world interventions. Furthermore, the shift from offline planning to a closed-loop policy allows the system to respond to environmental stochasticity in real-time.

Despite these advances, hurdles remain before clinical deployment. Like most causal methods, GIFT assumes there are no unobserved confounders—if a clinician makes a decision based on unrecorded intuition, the model could inherit bias. Additionally, the current study focuses on low-dimensional targets (e.g., tumor volume or a pair of vital signs), whereas real-world medicine often requires balancing dozens of competing metrics. Ultimately, GIFT is designed as a powerful decision-support tool, and future iterations will require stricter safety constraints and prospective clinical trials to ensure safe, effective patient care.

Summary updated

Abstract

Identifying optimal intervention sequences from offline data to guide temporal systems toward target outcomes is a critical challenge with profound implications for fields like personalized medicine. While existing methods are mostly evaluated in offline settings, practical applications demand online, adaptive strategies that can respond in real-time. To address this, we propose \textbf{G}oal-conditioned \textbf{I}ntervention via \textbf{F}actual-\textbf{T}argeted Training (\textbf{GIFT}), a novel framework for learning sequential intervention policies from observational data. GIFT learns a goal-conditioned policy by rescaling rewards with clipped importance weights, stabilizing learning and steering toward the target. Under standard assumptions, the induced operator has a unique fixed point and our procedure converges to it. We also bound the bias from clipping and approximation via the gap to the policy’s true value. Experiments show GIFT significantly outperforms existing methods in creating goal-conditioned policies for online deployment.
BibTeX
@inproceedings{wang2026target,
  author = {Xin Wang and Xiangyu Zhang and Shengfei Lyu and Huanhuan Chen},
  title = {{Target-Driven Policy Optimization for Sequential Counterfactual Outcome Control}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=2OtY6CfcFs},
}
Figure