When the Other Clinician Changes: Peer-Policy Drift in Retrospective Treatment Evaluation

Researchers increasingly use reinforcement learning on historical hospital records to ask questions like "how much IV fluid should a septic patient get?" These analyses quietly assume that everything except the treatment under study stayed fixed while the records were collected. Multi-agent learning has a name for what happens when that assumption fails: the moving target problem. Any one decision-maker's world looks unstable because other decision-makers keep changing how they act. In an intensive care unit, the value of a fluid strategy depends on how vasopressors (blood-pressure drugs) will be titrated in response to it. That titration practice is set by other people and shifts over the years a cohort spans. A pooled estimate of "the value of fluid policy X" from a decade of records is therefore an average against a moving target. Its answer depends on which era of co-treatment practice dominates the data.

We propose to make this dependence explicit and measurable. The method has three parts. It tests whether the policy governing a co-treatment was stationary across the cohort. It infers the co-treatment "regimes" from the co-treatment decisions themselves. It then evaluates a candidate policy for the treatment of interest against each regime separately, instead of against their historical mixture. The output is a value curve indexed by peer regime, not a single number. The central empirical question is whether the ranking of candidate treatment policies changes across co-treatment regimes. If it does, pooled retrospective evaluations are answering a question nobody asked.

Core Claim

Off-policy evaluation estimates the value of a new decision rule from data logged under old ones. In retrospective cohorts it has been treated as a single-agent problem. The behavior policy is assumed static, and any remaining bias is attributed to hidden patient-level confounding. We claim there is a second, distinct source of bias: drift in the policy of a co-acting decision-maker. It has a specific structure. It changes the co-treatment's response function, meaning how vasopressor dosing reacts to future patient states. Patient-level confounding instead changes who the patients are. Because the drift acts on how the co-treatment responds to future states, conditioning on the co-treatment's current action does not remove it. The fix we propose is to standardize evaluation to a stated reference co-treatment policy. This is the observational analogue of fixing the opponent's strategy, which AWESOME uses to make best responses well defined.

Write $\pi_A$ for a candidate policy for treatment A (fluids). Write $\mu_B^{(z)}$ for the logged co-treatment B policy (vasopressors) in regime $z$. Write $\pi_B^{(k)}$ for a chosen reference regime $k$. The quantity we estimate is

$$V(\pi_A; k) = \mathbb{E}_{\pi_A,\, \pi_B^{(k)}}\Big[\textstyle\sum_t \gamma^{\tau_t} r_t\Big],$$

where $r_t$ is the reward at decision $t$ and $\tau_t$ is elapsed clock time. The clock-time discount follows the semi-Markov correction in Semi-Markov Offline Reinforcement Learning for Healthcare. We estimate $V$ with per-decision importance weights $\pi_A/\mu_A$ for the A decisions and $\pi_B^{(k)}/\mu_B^{(z)}$ for the B decisions. The pooled estimator that everyone currently reports implicitly replaces $\pi_B^{(k)}$ with the historical mixture of all regimes. The new diagnostic quantity is the peer sensitivity $\max_k V(\pi_A;k) - \min_k V(\pi_A;k)$, together with any rank reversals among candidate $\pi_A$ across $k$.

Relation to Prior Work

Two clinical RL papers motivate the gap. Deep Reinforcement Learning for Sepsis Treatment pools fluids and vasopressors into one 25-action agent on 17,898 MIMIC-III sepsis patients. It treats the 2001–2012 behavior as stationary. Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare splits the value function additively across treatments. It acknowledges that the additive split is biased when the drugs interact. Neither paper asks whether the co-treatment's policy itself moved.

Two lines supply the method. Offline Reinforcement Learning from Datasets with Structured Non-Stationarity (COSPA) infers a latent parameter that drifts between episodes, using contrastive coding on trajectories. It does this for robot physics. We reuse the idea of inferring a between-episode latent, but apply it to the co-treatment's decisions, not to physics. The semi-Markov work above supplies the discounting needed for irregular charting.

Two papers constrain the design. Offline Policy Evaluation and Optimization Under Confounding clusters trajectories to recover persistent hidden patient traits. Applied naively, it would merge peer drift with patient type, and our ablation tests exactly that conflation. Offline Decentralized Multi-Agent Reinforcement Learning handles a related bias: teammates' bad logged actions make an agent's good actions look poor. It fixes this with optimistic reweighting in cooperative games. We deliberately avoid optimism. The semi-Markov paper reports roughly 20x value overestimation when offline clinical Q-learning is unconstrained, and that warns against importing it. We also do not claim that clinicians are reward-sharing learners, which is the assumption of the lenient methods in Lenient Multi-Agent Deep Reinforcement Learning. We only require that their logged policy changed.

Research Program

The first component is a stationarity test for the co-treatment policy. It takes AWESOME's "peers are playing stationary strategies" hypothesis test from repeated games and moves it to observational records. We fit a state-conditional model of B's actions. We then compare held-out likelihood under a single model against a model with $K$ latent regimes, calibrating the test with a permutation null over stays.

The second component infers regimes. A contrastive encoder in the style of COSPA is trained only on (state, B-action, elapsed-time) segments of each stay. Restricting it to B's behavior, not full trajectories, is the key ingredient. It is meant to pick up differences in how B responds to a given state, not differences in who the patients are.

The third component is regime-standardized evaluation as defined above. We report bootstrap intervals for $V(\pi_A;k)$ and for the peer-sensitivity gap. A secondary step learns a fluid policy that maximizes worst-case value across inferred regimes, borrowing the population view of Policy Space Response Oracles: A Survey. This is secondary because the diagnostic is the contribution.

We are direct about identification. On real records, a "regime" inferred from B's behavior cannot be fully separated from unmeasured severity without an external era signal. MIMIC-III's random date-shifting (MIMIC-III, a freely accessible critical care database) removes calendar time, so we need a proxy. We use the database's two charting systems, which roughly separate earlier from later years, as an era marker. We validate inferred regimes against it. The Warfarin cohort's four constituent trials give a second natural regime label: trial identity, with concomitant antithrombotic co-therapy as the B treatment where it is recorded. Ground truth for identification comes only from a simulator.

Baselines

The naive reference point is pooled semi-Markov importance-sampling evaluation, which treats the behavior policy as stationary. This is what current clinical off-policy studies effectively report. It runs on identical cohorts, features and candidate policies. Beating it means lower absolute evaluation error against simulator ground truth. It also means correctly detecting rank reversals that the pooled estimate hides.

The strongest simple competitor is era stratification: estimating values separately within the known era label (simulator era, charting-system era, trial identity) using the same estimator. When the era label is clean and drift happens only between eras, this should match our method. Our method must beat it in two settings. One is when drift occurs within eras, for example when practice changes partway through an era. The other is when the label is coarse. In both settings the inferred regimes should give lower evaluation error.

The strongest existing method for latent structure is the trajectory-clustering estimator of Offline Policy Evaluation and Optimization Under Confounding. We run it with its own clustering and the same number of clusters and tuning budget, then plug its clusters into the regime-standardized estimator. No reported result found on Lacuna for its evaluation error in numeric form, so we rerun it.

For policy learning we compare against the factored-action Q-learner of Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare and the pooled D3QN of Raghu et al. Both are trained with the same hyperparameter search budget. No reported numeric MIMIC-III policy return was found on Lacuna for either; the file reports only qualitative findings, so we report our own reruns.

The ablation that isolates the key ingredient is the same contrastive encoder trained on full trajectories, including patient states and A-actions, instead of B-behavior only. If the full-trajectory encoder recovers regimes equally well when a hidden patient confounder is present, our central design choice is unnecessary.

Why not just…?

Why not just use propensity score matching with a Cox model? Propensity matching, as in Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach, balances baseline covariates for a single exposure at diagnosis. Fluid and vasopressor dosing are repeated decisions that react to each other within hours, and a point-exposure design has no object corresponding to "the co-treatment's response policy." Matching also adjusts only for observed covariates. We nonetheless include it on a point-exposure version of the question (high versus low first-day fluid), stratified by era, as a sanity check that our estimates agree where both methods apply.

Why not just stratify by calendar year or era? This is the real threat. It is exactly what an epidemiologist would do with NCDB-style drift, where chemotherapy use fell from 25.8% to 18.4% over 2010–2013 (Practice patterns and outcomes for patients with node-negative hormone receptor-positive breast cancer and intermediate 21-gene Recurrence Scores). In MIMIC-III, calendar time is destroyed by date-shifting, leaving only a coarse proxy. We also expect drift that is gradual or tied to specific units not to line up with era boundaries. If the simulator experiments show that era stratification matches inferred regimes under realistic within-era drift, we would retreat to recommending stratification plus our stationarity test. That would still change how clinical off-policy results are reported.

Why not just put the co-treatment into the state or the joint action? Conditioning on B's current dose does not fix B's future reaction to the states that A's choices create, and that future response function is what drifts. Evaluating a joint policy is legitimate, but it requires specifying B. The pooled joint model silently specifies B as the historical mixture. The simulator experiment tests this directly. A centralized model with the co-treatment in the state should show the same rank reversals as pooled evaluation when B's response function changes between eras.

Proposed Experiments

The decisive experiment uses the Sepsis Simulator used by both the factored-action and confounding papers. A is vasopressor dosing. B is the antibiotic and ventilation policy, whose response to blood pressure shifts between two eras, and we vary the size of that shift. We also inject the persistent diabetes confounder used by Kausik et al., so that peer drift and patient confounding coexist and true values are known for every regime. We test three things. First, whether the stationarity test holds nominal false-positive rate when there is no drift and gains power as drift grows. Second, whether B-only regime inference beats the full-trajectory ablation at recovering true regimes, measured by adjusted Rand index, in the presence of the confounder. Third, whether regime-standardized evaluation reduces absolute value error and recovers the true ranking of five candidate A-policies, measured by Spearman correlation per regime, when pooled evaluation and era stratification do not.

On MIMIC-III we rebuild the Raghu sepsis cohort with semi-Markov timing. We test vasopressor-policy stationarity. We check whether inferred regimes align with the charting-system era beyond what patient acuity (SOFA, the standard organ-failure severity score) explains. We then report peer-sensitivity gaps for the published-style fluid policies. On the Warfarin multi-trial cohort we ask whether SBCQ's dosing-policy value is stable across trials.

The result that would sink the proposal is clear in advance. It would be small peer-sensitivity gaps and no rank reversals on real data, even though the simulator shows the method can detect them. A positive result would be different: a fluid policy judged best by pooled evaluation that is not best under the later vasopressor regime, with the gap exceeding bootstrap uncertainty. That result would mean retrospective treatment-policy claims must be reported conditional on co-treatment practice.