When the Other Clinicians Change: Co-Treatment Drift in Retrospective Cohort Estimates

A retrospective cohort study estimates "the effect of treatment A" from historical hospital records. In practice, A is never given alone. The effect of intubating a COVID-19 patient a day earlier depends on what the rest of the care team does around that decision, such as steroids, proning and fluid management. Those surrounding practices change from year to year and from hospital to hospital. Multi-agent reinforcement learning, the study of several decision-makers learning in a shared environment, has a precise name for this problem. When one agent treats its co-actors as part of the environment, the outcomes it sees are averaged over the co-actors' current behavior. Once that behavior shifts, the apparent dynamics shift with it and old data becomes misleading. We argue that a published cohort effect has the same structure. It is an average over the co-treatment habits of the clinicians who happened to be practicing during the study window, and it goes stale when those habits change.

The proposal makes this dependence explicit and measurable. We estimate the outcome as a function of the target treatment and the concurrent co-treatments jointly, which is the cohort-study analogue of the "centralized critic" that restores stationarity in multi-agent learning. We then re-average under a stated target co-treatment policy, such as current practice or another hospital's practice. On top of this we build a decomposition. It splits the change in an effect estimate across eras or sites into three parts: co-treatment drift, case-mix drift, and residual change in the underlying biology or mechanism. The same lens extends to offline reinforcement learning for a single treatment lever, whose value depends on how the clinicians running the other levers behave.

Core Claim

From the perspective of agent $i$, the transition law in multi-agent learning is $P(s' \mid s, a_i) = \sum_{\mathbf{a}_{-i}} P(s' \mid s, a_i, \mathbf{a}_{-i}) \prod_{j \neq i} \pi_j(a_j \mid s)$. Here $s$ is the state, $a_i$ is agent $i$'s action, $\mathbf{a}_{-i}$ are the other agents' actions, and $\pi_j$ are their policies. Read clinically, $s$ is the patient's state, $a_i$ is the treatment under study, $\mathbf{a}_{-i}$ are the concurrent co-treatments, and $\pi_j$ are the co-treatment policies of the rest of the care team. A cohort effect estimated with propensity weighting on $a_i$ alone targets

$$\tau^{e} = \mathbb{E}_{s \sim p^{e}}\Big[\sum_{\mathbf{a}_{-i}} \big(\mu(s, 1, \mathbf{a}_{-i}) - \mu(s, 0, \mathbf{a}_{-i})\big)\, \pi^{e}_{-i}(\mathbf{a}_{-i} \mid s, a_i)\Big],$$

where $e$ indexes an era or site, $p^{e}$ is that era's case mix, $\mu$ is the expected outcome given state, target treatment and co-treatments, and $\pi^{e}_{-i}$ is that era's co-treatment policy.

The claim is threefold. First, $\tau^e$ moves whenever $\pi^e_{-i}$ moves, even if $\mu$ is fixed. Second, retrospective designs, including federated ones, implicitly assume $\pi_{-i}$ is stationary. Third, separating $\mu$ from $\pi^e_{-i}$ lets us report effects under an explicitly chosen co-policy and attribute observed cross-era disagreement to its sources.

We are explicit about what is not new. The estimator at the core is the g-formula with a stochastic intervention on co-treatments, which is standard causal-inference machinery. The contribution is to identify the co-treatment policy as the drifting quantity, borrowing the non-stationarity analysis from multi-agent learning. On that identification we build a falsifiable drift decomposition and a co-policy-shift evaluation for offline RL, and we test both against ground truth.

Grounding in Prior Work

Several retrospective studies motivate the gap because they report marginal effects inside a moving practice environment. One analysis of the National Cancer Database regresses chemotherapy receipt on clinical predictors in intermediate-risk breast cancer. It finds no 5-year survival difference (97.4% vs 97.8%) while documenting chemotherapy use falling from 25.8% to 18.4% over 2010–2013 (Practice patterns and outcomes…). That is exactly the kind of practice shift that changes what any marginal contrast averages over. A single-center Cox analysis estimates a per-day intubation-delay hazard ratio of 1.018 (95% CI 1.005–1.03) (Timing to intubation COVID-19 patients). This number is defined only relative to that hospital's co-management during that pandemic phase. A propensity-score comparison of prostate cancer treatments spans 1980–1997 (Long-Term Survival in Men With High Grade Prostate Cancer), a window over which supportive care certainly changed. We do not dispute these estimates. We argue that they need a co-policy label.

Three papers supply methods and constraints. Fed-IPW/Fed-AIPW estimate global propensity scores across 14 trauma centers without pooling data, and they handle sites that never treat certain subgroups (Federated Causal Inference…). The paper explicitly leaves site-level differences unexplained and stays single-step. We treat between-site variation in co-treatment as a candidate explanation and as useful variation. Delphic offline RL shows that hidden confounding can leave several causal "worlds" equally compatible with data, and it penalizes value estimates by disagreement among them (Delphic Offline RL). Co-treatments are themselves confounded by unrecorded bedside judgment, so we reuse its ensemble-of-compatible-worlds idea to bound $\mu$ rather than point-estimate it. ComaDICE (ComaDICE) and agent-specific conservatism (Who Matters Matters) show that joint-action combinations are sparsely covered in offline multi-agent data and that pessimism should differ by actor. Both assume a single stationary logging policy and pure cooperation. In our setting the logging co-policy is the very thing that drifts, so those methods are a constraint on what is estimable rather than a method we copy.

The multi-agent foundations supply the conceptual mechanism. Independent learners confuse teammates' policy changes with environmental change (Claus & Boutilier; Predicting and Preventing Coordination Problems). Conditioning on the joint action during training restores stationarity (MADDPG), and counterfactual baselines that hold co-actors fixed isolate one actor's contribution (COMA). Our $\mu(s, a_i, \mathbf{a}_{-i})$ plays the role of that joint-action critic, but it is fit on observational data.

Finally, the closest clinical RL work, D3QN on 17,898 MIMIC-III sepsis patients (Deep RL for Sepsis Treatment), collapses fluids and vasopressors into one 25-action grid. It diverges from clinicians in high-severity patients. Our angle differs from simply "splitting that grid into two agents." A 25-action space is not a combinatorial explosion, and a factored single-agent model would handle it. The real multi-agent issue is that a deployed tool controls one lever, while humans controlling the other levers keep changing their behavior.

Research Program

The first component is estimation. For a chosen target treatment and its co-treatments, we fit the outcome model $\mu$ with doubly robust, cross-fitted estimation over a time-discretized trajectory, using 4-hour windows following the sepsis cohort convention. We fit era- or site-specific co-policies $\pi^e_{-i}$ as conditional classifiers. Co-treatments are time-varying and affected by earlier target treatment, so we use the sequential g-formula rather than covariate adjustment. Where joint coverage of $(a_i, \mathbf{a}_{-i})$ is thin, we restrict the target co-policy to the supported region. We report the fraction of mass removed instead of extrapolating. To address hidden confounding, we train a small ensemble of outcome models compatible with the observational data, in the Delphic style. We report effect intervals across that ensemble.

The second component is the drift decomposition. For two eras $e, e'$, we write $\tau^{e'} - \tau^{e}$ as a telescoping sum. One term swaps the co-policy $\pi_{-i}^{e} \to \pi_{-i}^{e'}$ with case mix and mechanism held fixed. A second term swaps the case mix $p^e \to p^{e'}$. The remainder is change in $\mu$, meaning a mechanism change or something unmodeled. We use both orderings and report the average to avoid order dependence. The remainder term makes the method falsifiable: if co-treatment drift is a real driver, the remainder should be small relative to the co-policy term in eras where practice is known to have shifted.

The third component is offline RL under co-policy shift. Policies are learned for one lever, for example vasopressors, while another lever, fluids, stays under human control. We evaluate them with the learned $\mu$-based model under the logged co-policy of a different era, not the training era. We then compare three approaches on this evaluation. The first is a policy trained conditional on co-treatment, so it acts on observed fluids. The second is an independent learner that absorbs fluids into the environment. The third is the joint-grid D3QN.

Baselines

The naive reference point is a pooled marginal estimator: AIPW on the target treatment across all eras and sites, ignoring co-treatments except as baseline covariates. This is what most retrospective cohorts report. Our method beats it if, on held-out eras or sites, its error against ground truth or its predictive miscalibration is smaller. It must also not widen intervals so much that coverage gains are trivial.

The strongest existing method for multi-site observational effects is Fed-AIPW (Federated Causal Inference…). That paper reports it matching a pooled benchmark with lower variance than meta-analysis, but the file gives no numerical effect or error value (no reported result found on Lacuna for Fed-AIPW error on Traumabase). We run it on the same patients, covariates and folds. Its outcome and propensity models get the same hyperparameter search budget as ours. Our method counts as beating it if it predicts held-out-site effects with lower error, specifically in sites whose co-treatment policy differs from the training sites.

The simple competitor we consider most dangerous is era-restricted estimation: using only the most recent era. The clinical-RL comparator is the joint-grid D3QN of Deep RL for Sepsis Treatment, which reports no numerical policy return (no reported result found on Lacuna for its return). For a sense of how hard outcome modeling is in this data, MIMIC-Extract reports mortality-prediction AUROC between 0.85 and 0.90, and our $\mu$ should be in that range before any effect is trusted.

The key ablation removes re-averaging. In that variant we fit $\mu$ jointly but average over the pooled observed co-treatments. The difference between this ablation and the full method isolates the value of naming a target co-policy.

Why not just…?

Why not just adjust for co-treatments as covariates in a Cox or propensity model? Conditioning on co-treatments gives an effect at whatever co-treatment mixture the data happened to contain, not under a chosen target. When co-treatments respond to earlier target treatment, adjusting for them introduces bias, which is the standard treatment–confounder feedback problem. The covariate-adjustment arm is included in the simulator experiment, where ground truth exposes that bias directly.

Why not just estimate within the most recent era? This is a real threat when the recent era is large and the future resembles it. It halves or worse the sample. That matters for rare subgroups, which the multicenter literature exists to reach, and it still assumes stationarity going forward. We include it as a baseline. If it matches our method on held-out-era error with comparable interval width, we will report that co-policy modeling adds nothing in that setting.

Why not just use a joint-action single-agent RL model? A joint model is correct if the tool controls every lever. Deployed decision support usually controls one lever while humans keep running the others, and the humans adapt. The co-policy-shift evaluation tests directly whether joint-grid D3QN degrades more than a co-treatment-conditioned policy when the human co-policy changes.

Why not just use independent per-treatment learners? They are the clinical analogue of independent learners. That design shows coordination failure in simulation, with independent actor-critic below 70% win rate against roughly 81% mean for COMA on SMAC 5m (COMA). That evidence comes from games, so we test the clinical version in the shifted-co-policy evaluation rather than assume it transfers.

Proposed Experiments

The decisive test is semi-synthetic, because only there is the true effect known. In the sepsis simulator used by Delphic ORL and DTR-Bench's OberstSepsisEnv (DTR-Bench), we designate one intervention as the target and the others as co-treatments. We generate synthetic "eras" whose co-treatment logging policies drift by controlled amounts. We also keep the hidden diabetes confounder at graded strengths. Ground-truth $\tau^e$ under any co-policy is computable by rollout. We measure absolute error of each estimator for a held-out target co-policy, interval coverage, and recovery of the known shares in the drift decomposition. For the RL arm, we measure true return of each learned policy under a shifted co-policy. The prediction is specific: pooled and Fed-AIPW-style estimators should have error that grows with co-policy drift, our method's error should stay flat, and the gap should narrow to zero when drift is zero. If the gap does not appear, the core claim fails.

On real data we cannot observe truth, so we use temporal and site hold-out as a predictive check. In the MIMIC-III sepsis cohort, we split by recording era; the change of bedside documentation system partway through the database is a plausible proxy, to be confirmed during extraction. The target is vasopressors and the co-treatment is IV fluids. We fit on the early era, re-average under the late era's estimated co-policy and case mix, and compare against the late era's own doubly robust estimate. The same is done on HiRID, which reports no numerical metric for Delphic ORL on Lacuna. If Traumabase access is possible, we treat its 14 centers as co-policy variation and transport tranexamic-acid effects to a held-out center. We report each held-out discrepancy alongside the decomposition. A large remainder term on real data is an informative negative result: it says practice drift is not the main reason cohort estimates disagree.