Prognosis and Treatment Effect as the Two Streams of a Dueling Network

Deep reinforcement learning has been used to recommend ICU treatments from historical records, and it fails in a telling way. A recommender trained on sepsis patients was 5 to 7 times more likely than physicians to give vasopressors to low-risk patients. Policies trained from different random seeds scored the same yet recommended very different doses. We argue that the architecture these systems use, the dueling network, already has the right two slots for clinical heterogeneity but fills them with the wrong quantities. A dueling network splits its estimate of "how good is action $a$ in state $s$" into a scalar for how good the state is regardless of action and a per-action correction. In medicine these two pieces should be the patient's prognosis under usual care and the patient-specific benefit of departing from usual care. Disease progression modeling has spent a decade showing that prognosis depends on two hidden quantities: which biological pathway, or subtype, a patient follows, and how far along it they are, the stage. Treatment response plausibly depends on the same two.

The proposal is to anchor the dueling split so that each stream is identified from observational data. The state stream is defined as the expected return under the clinicians' own policy, which can be estimated without ever trying an untested action. The action stream is then a treatment-effect term that varies with an inferred subtype and stage, and it is pulled toward zero wherever the data contain no evidence. The prediction is that this removes the low-risk over-aggression and the seed instability without lowering estimated outcomes. Because the state stream now means "prognosis under usual care," its output can also be checked directly against progression-model forecasts. No existing clinical dueling network offers that kind of external validity check.

What exists and where the gap is

Three clinical systems use dueling networks on ICU data. Raghu et al. compress 47 MIMIC-III sepsis variables with a sparse autoencoder and train a dueling double DQN over a flattened 5×5 grid of fluid and vasopressor doses. DeepEN does the same for enteral nutrition on MIMIC-IV and adds Conservative Q-Learning, a penalty on values assigned to actions rarely seen in the data. In both, the dueling split is an optimization convenience inherited from Wang et al.. The advantages are centered by subtracting their uniform mean over all 25 actions, so the state stream's value depends on advantages of doses clinicians almost never give. Neither system represents subtypes or stages; all patients share one undifferentiated latent state. Lu et al. supply the motivating evidence. Their sensitivity analysis of the same architecture found the low-risk vasopressor over-aggression, a 10.7% standard deviation in vasopressor recommendations across seeds, and dose inflation of up to 1.8× for vasopressors and 3.1× for fluids when treatment history was dropped. They proposed no architectural fix.

On the heterogeneity side, the methods supply the inputs we need. Missing-data SuStaIn infers subtype and stage jointly from cross-sectional snapshots. It handles absent measurements by giving them a uniform likelihood instead of imputing them, and it found five ADNI subtypes with very different conversion rates. Its central lesson is that "moderately abnormal" can mean early in a fast subtype or late in a slow one, and clustering on severity confuses the two. A sepsis analogue is exactly what a SOFA-score risk stratum hides. The multi-resolution trajectory framework shows that population, subpopulation and individual layers improve forecasts: 14.3% lower MAE and a 31% versus 17% true-positive rate for critical lung-function drops. That supports making prognosis subtype-aware. Coordinatewise-Soft-Impute is the closest progression model that handles treatment at all. It separates natural trajectories from a single additive treatment effect shared by every patient and assumes treatment is assigned independently of disease type. It constrains our design in a useful way: the effect it estimates is exactly the quantity our action stream should make subtype-specific and endogenous, and it has no notion of sequential policy.

So the obvious combination, appending a subtype label to a D3QN's input, is not claimed here as new. What is new is the claim that the dueling identifiability constraint, chosen for numerical stability on Atari, is the wrong constraint offline. Replacing it with a behavior-anchored, subtype-structured constraint is what makes the two streams mean prognosis and heterogeneous treatment effect.

Core Claim

Let $h$ be a patient's history and $s$ its encoded state. Let $\hat\pi_b(a\mid s)$ be a behavior-cloned model of clinician dosing. Let $q_k(h)$ be the posterior probability, from a SuStaIn-style model fit beforehand on physiology snapshots, that the patient belongs to subtype $k$ of $K$, with the inferred stage included in $s$. We define

$$Q(s,a) = V(s) + \sum_{k=1}^{K} q_k(h)\Big[A_k(s,a) - \sum_{a'} \hat\pi_b(a'\mid s)\,A_k(s,a')\Big],$$

where $V$ is a scalar head and each $A_k$ is a subtype-specific advantage head. Because the advantages are centered under the clinician policy, not the uniform action distribution, $V(s)$ is the expected return of usual care. It is trained with on-data temporal-difference targets that never query untested actions, which makes it an observable prognosis. Each $A_k$ is shrunk toward a shared head $A_0$, and all of them toward zero, with a penalty weighted by $1/\hat\pi_b(a\mid s)$. Departures from usual care therefore earn credit only where the data, within that subtype, show them. Following action branching, the 5×5 grid is factored into a fluid branch and a vasopressor branch sharing $V$. This lets a subtype's vasopressor effect be estimated without being diluted across 25 joint cells.

The hypothesis is mechanistic. Over-aggression in low-risk patients arises because, under uniform centering, prognostic variance leaks into advantages of rarely-observed high doses, and low-SOFA patients are where high doses are rarest. Seed instability is the same unidentified mass moving around. Behavior-anchored centering removes the leak, and subtype structure lets genuine benefit survive where pooled shrinkage would erase it.

Research Program

The work has three stages. First, we fit missing-data SuStaIn to cross-sectional organ-dysfunction markers at each 4-hour window: lactate, creatinine, bilirubin, platelets, mean arterial pressure and the remaining SOFA components. Missing labs are handled through the uniform likelihood instead of imputation. We choose $K$ by the usual cross-validated likelihood and report whether the inferred stage separates patients that SOFA lumps together. Second, we build the anchored dueling network above on the sepsis cohort definition and recurrent encoder of Lu et al., including treatment history, since its exclusion is a known failure. The network is trained with double-Q targets. Third, we analyze the streams. We test whether $V(s)$, read as predicted survival under usual care, is calibrated against observed outcomes for each subtype and stage, and whether it matches a subtype-aware progression forecast. We test whether the learned subtype advantages disagree, for instance whether vasopressor advantage is concentrated in one subtype.

The synthetic stage carries the strongest claims, because offline evaluation on real ICU data cannot establish ground truth. We extend the CSI mixture-Gaussian simulator so that each latent subtype has its own treatment effect. Clinicians in the simulation treat as a function of the latent subtype, which makes treatment endogenous, exactly the case CSI excludes. The simulation also includes a low-risk subtype for which aggressive dosing is harmful but rarely observed. True optimal policies and true per-subtype effects are then known.

Baselines

The naive reference is the observed clinician policy, approximated by the behavior-cloned $\hat\pi_b$. On MIMIC-III, Raghu et al. report an observed clinician mortality of 13.7%; on MIMIC-IV enteral nutrition, DeepEN reports 22.5%. Any learned policy must match its estimated value while dosing more conservatively in low-risk patients. Otherwise it is not worth deploying.

The strongest existing method is a dueling double DQN with Conservative Q-Learning, as in DeepEN, given the same encoder, treatment history and intermediate SOFA-based rewards. DeepEN reports an estimated mortality of 18.8% against 22.5% for clinicians and a CWPDIS return of 11.89 against 5.87; Raghu et al. report 11.2% ± 0.4% estimated mortality for their dueling Q-network. These are the reference figures, though our real-data runs will be on our own cohorts and will report reproduced, not copied, values. To make it as strong as possible, this baseline also receives the SuStaIn subtype posteriors and stage as extra input features. Both systems get the same data splits, the same number of hyperparameter configurations (we budget 30 each, including the CQL weight and our shrinkage weight), and the same 10 seeds. Beating it means that, at a statistically indistinguishable or better off-policy return, we reduce the low-risk vasopressor rate ratio relative to physicians well below the 5 to 7× that Lu et al. found for unconstrained Duel-DDQN, and we roughly halve the cross-seed dosing standard deviation relative to the 10.7% they report. In simulation, beating it means lower true regret and lower error in recovering subtype-specific effects.

Two ablations isolate the ingredients. The first keeps uniform mean centering but retains subtype heads, which tests whether the anchoring matters. The second keeps behavior centering but uses a single pooled advantage head, which tests whether subtype structure matters beyond a better constraint. A third variant replaces SuStaIn posteriors with k-means clusters on the same snapshot features, to test whether stage-aware subtyping matters or any partition would do.

Why not just…?

Why not just feed subtype labels into an existing D3QN with CQL? This is the most serious competitor, and it is a baseline above for that reason. The concern is that conditioning the input does not change what the streams are identified as. Under uniform centering, the advantages of rarely-taken actions remain anchored to nothing, and CQL suppresses them by a single global weight that cannot tell "never tried in this subtype" from "harmful in this subtype." If this baseline matches our stability and low-risk numbers in both simulation and MIMIC, the architectural claim is wrong, and we will report it that way.

Why not just train a separate policy per subtype? The ADNI cohort had 789 participants across five subtypes; sepsis subtypes will likewise leave some small groups. Separate policies throw away shared prognostic structure, and the smallest subtypes, which may be the low-risk ones where errors are dangerous, get the least data. The shrinkage of each $A_k$ toward $A_0$ is the multi-resolution idea of Alaa-style hierarchical trajectory models applied to treatment effects. We include per-subtype policies as an extra comparison in simulation, where small-subtype regret can be measured.

Why not just stratify by SOFA and constrain doses in low-SOFA patients? A hand rule would fix the specific symptom Lu et al. measured without addressing the cause, and it bakes in the severity-based grouping that SuStaIn was designed to undo. The simulator includes a subtype that looks mild at a snapshot but is early in a fast trajectory and benefits from early escalation. A SOFA cap should under-treat exactly those patients, and our method should not. If the cap matches us in simulation, heterogeneity is not doing work and we would say so.

Proposed Experiments

The decisive checks come in a fixed order. In the endogenous-treatment simulator, the anchored subtype dueling network must recover each subtype's true treatment effect, measured as error in per-subtype advantage. It must also achieve lower true regret than the CQL baseline with subtype features, with the gap growing as treatment assignment becomes more subtype-dependent. If the gap does not grow, the mechanism is not the one we claim. On the MIMIC-III sepsis cohort of Lu et al., we measure the low-risk vasopressor rate ratio against physicians and the cross-seed standard deviation of recommended doses. We report off-policy return with weighted doubly robust and CWPDIS estimators only as a non-inferiority check, since Lu et al. showed these estimators cannot tell divergent policies apart. On MIMIC-IV enteral nutrition, we repeat the analysis against DeepEN's setting to test whether the effect transfers across interventions. Across all settings, we test calibration of $V(s)$ as usual-care survival within each subtype and stage. If the value stream is miscalibrated within subtypes, the claim that it can be read as prognosis fails, even if the policy metrics improve.