How Much Does Uniformity Cost? Measuring Protocol Gaps as Jensen Gaps in Belief Space

A clinical guideline such as "treat every patient's infection for seven days" is a single fixed decision rule applied to everyone. In the language of planning under uncertainty, it is one policy whose expected value depends linearly on what we believe about the patient's hidden condition. The best personalized policy instead picks a different rule for each belief, and its value is the upper envelope of many such lines. Planning theory has long established that this envelope is convex. A convex curve sits above every line that touches it, and the gap between them is zero only at the touching point. That point is the "average patient" the guideline was designed for.

This note proposes to make that picture quantitative. The cost of a uniform protocol over a patient population can be computed as a Jensen gap: the average of a convex function minus the function evaluated at the average. That gap can be split into three parts. One comes from personalizing on baseline risk, one from adapting to bedside observations, and one from the residual optimal policy. Because the envelope is a finite set of lines, the same geometry yields an algorithm for the question guideline committees actually ask: how many patient strata does a protocol need, and which rule goes with each? We will test this on ICU antibiotic discontinuation in MIMIC-III. Fixed treatment durations are the standard there, and a hidden two-or-three-state infection status keeps exact belief-space computation tractable.

Core Claim

A belief state is the probability distribution an agent keeps over hidden states it cannot observe directly, here whether a patient's infection has resolved. A partially observable Markov decision process (POMDP) is the planning model built on such beliefs.

In a POMDP, any fixed policy, including one that branches on observations, has value $V_\pi(b)=\langle \alpha_\pi, b\rangle$. Here $b$ is the patient's belief vector and $\alpha_\pi$ is a vector of per-hidden-state values called an α-vector. The optimal value $V^*(b)=\max_\pi \langle\alpha_\pi,b\rangle$ is piecewise linear and convex. Araya et al. show that convexity survives when the reward itself depends on the belief, provided that reward is convex (A POMDP Extension with Belief-dependent Rewards). An example is a term rewarding confidence at the moment of stopping treatment.

Let patients $i$ have baseline beliefs $b_i$ with mean $\bar b$. The value lost by the best single protocol, which we call the personalization gain, is

$$G \;=\; \mathbb{E}_i\big[V^*(b_i)\big] \;-\; \max_\pi \langle \alpha_\pi, \bar b\rangle \;=\; \mathbb{E}_i\big[V^*(b_i)\big] - V^*(\bar b) \;\ge\; 0,$$

where the equality holds because a single policy's value is linear in belief. $G$ is exactly the Jensen gap of $V^*$ over the population's spread of beliefs. It therefore grows with how dispersed patients are and how curved the envelope is near $\bar b$.

The best protocol with $K$ strata uses a set $S$ of $K$ policies, each patient getting the best one in the set:

$$G_K \;=\; \mathbb{E}_i\big[V^*(b_i)\big] - \max_{|S|=K}\, \mathbb{E}_i\big[\max_{\pi\in S}\langle\alpha_\pi, b_i\rangle\big].$$

The inner objective is a facility-location function of $S$, which is monotone submodular. Greedy selection of policies is therefore within a factor $(1-1/e)$ of the best $K$-stratum protocol. This gives a principled guideline-compression procedure, and the curve $G_K$ against $K$ shows directly whether a disease needs two strata or twenty.

The novelty is not a new clinical RL policy. It is an estimator and a decomposition of what uniformity costs, built from the convex geometry of belief space, together with a method for compressing the personalized optimum into guideline-sized rules. The decomposition makes the motivating failure mode concrete: low-risk patients overtreated and high-risk patients undertreated under one rule. Clinical studies document this pattern for the fixed three-month anticoagulation window (Risk factors for delayed recanalization of calf vein thrombosis) and for blanket conservative PCOS stimulation protocols (Follicular-Phase GnRH Agonist Protocol…). These papers motivate the question but supply no method.

Research Program

Model. On MIMIC-III (MIMIC-III, a freely accessible critical care database), we fit a small input-output hidden Markov model of therapeutic antibiotic courses.

  • Hidden state: infection active, resolving, or resolved.
  • Actions: continue or stop the antibiotic, once per day.
  • Observations: temperature, white cell count, lactate, and vasopressor need, each with explicit missingness indicators.
  • Outcomes: reinitiation within a window, and 90-day mortality.

The outcome choice follows Explainable machine learning for discontinuation of therapeutic antibiotics in intensive care patients. That study found short final courses to be the most consistent predictor of reinitiation, and restarting the same drug to be associated with higher 90-day mortality. Both outcomes are therefore clinically meaningful cost terms.

Baseline beliefs $b_i$ come from admission covariates through the model's initial-state distribution. The small hidden state is deliberate. It sidesteps the double-exponential growth of α-vectors that dominates the belief-planning literature, so exact or near-exact value iteration is feasible and $V^*$ is the true optimum of the fitted model rather than an approximation.

Decomposition. We compute four values under the fitted model.

  1. Best open-loop fixed duration, a single α-vector with no observation branches.
  2. Best closed-loop uniform rule, a single finite controller that branches on observations, such as "stop after day 5 if afebrile."
  3. Best $K$-stratum protocol, built greedily for $K=1,\dots,10$.
  4. Fully belief-adaptive optimum.

The differences between consecutive tiers separate three sources of gain: within-trajectory adaptation, baseline-risk personalization, and the residual. We will also add Araya's convex belief-dependent term that rewards certainty at stopping, and report how sensitive $G$ is to its weight. Each selected α-vector corresponds to a readable policy graph, so the $K$-stratum output is a set of decision rules a clinician can audit.

Controlled check. The theory needs a setting with known ground truth. We generate synthetic POMDPs with controlled envelope curvature and controlled patient-belief dispersion. This tests three things: that the estimated $G$ matches the true one, that greedy compression stays near the exhaustive optimum on small instances, and how much model misspecification biases $G$.

Baselines

Uniform guideline (the naive reference). The naive reference point is the actual uniform guideline: the fixed course length most common in the cohort, evaluated under the same fitted model and in held-out data. Any proposal must first show a nonzero, statistically resolvable $G$ against it. If $G$'s confidence interval covers zero, uniformity is justified for this decision and that is itself the finding.

Static stratified rule. The strongest competitor on the actual question is a stratified fixed-duration rule. Patients are grouped by a baseline risk model (a gradient-boosted reinitiation predictor trained on the same features), and each group gets its best fixed duration. This is how guidelines already personalize, and it is the "static risk score" approach that DPSCREEN argued against. We give it the same number of strata $K$, the same features, and the same model-selection budget as the greedy α-vector compression. It beats us if its value equals ours at matched $K$.

Dynamic belief-space personalization. DPSCREEN is the closest existing strong result for belief-space personalization against a uniform schedule. It used point-based value iteration on a screening POMDP and reports reducing unnecessary screenings by 28% to 68% at the same expected detection delay as annual screening (DPSCREEN: Dynamic Personalized Screening). We found no reported result on Lacuna for POMDP or RL-based antibiotic discontinuation in MIMIC-III, so there is no published number to beat on our primary task. We will instead run DPSCREEN's point-based solver on our model as the approximate-solver baseline.

Learned recurrent policy. We train a recurrent offline Q-learning policy in the style of Deep Reinforcement Learning for Sepsis Treatment on the same cohort, with matched tuning budget.

Ablations. Removing observation branching (tier 1 versus tier 2) isolates the adaptation ingredient. Replacing greedy facility-location selection with random or k-means-on-belief strata isolates the compression ingredient.

What counts as success. We claim success if two conditions hold. First, the greedy α-vector protocol at small $K$, around three, recovers most of the fully personalized gain. Second, it beats the risk-stratified rule at matched $K$, both under the fitted model and on held-out observational checks.

Why not just…?

Why not just train a recurrent offline RL policy and compare it to the guideline with importance sampling? Bradford et al. showed that importance-sampling estimates in ICU RL can be inflated by recommending inaction for stable patients (Does the "Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care?). Our setting has a direct analogue of that "zero drug" problem: stopping early looks cheap on patients who were never really infected. A recurrent policy also yields one number and one black box, with no decomposition of where the gain comes from and no guideline-sized rule. We still run it as a baseline. If it agrees with our model-based $G$, that is corroboration. If it reports much larger gains, we will check whether those gains concentrate in low-belief patients, which is the signature of metric gaming.

Why not just stratify by a baseline risk score? This is the real threat, and we treat it as a hypothesis rather than a strawman. If baseline dispersion of beliefs explains almost all of $G$, then tier 3 at small $K$ will nearly equal a risk-stratified rule. The honest conclusion would then be that existing risk-score practice is close to sufficient. The decomposition is designed to report exactly this. Our proposal adds value only where tiers 2 and 4 open a gap, or where α-vector selection chooses better strata than risk quantiles.

Why not just simulate the fitted model by Monte Carlo and compare policies? For estimating any single policy's value, simulation is equivalent, and we use it for verification. What simulation does not give is the search over protocols. The convex structure turns "find the best $K$-rule guideline" into submodular maximization over α-vectors with a guarantee. It also explains why gains appear: patients far from the tangent point of the uniform rule account for most of the loss.

Why not just a myopic stop-when-posterior-is-low threshold? A myopic threshold is a single closed-loop controller, so it is one of the candidates in tier 2. Araya et al. argue that greedy information use is suboptimal over longer horizons. The tier 2 versus tier 4 comparison measures how much that matters here, and a negligible difference would be reported as such.

Proposed Experiments and Decisive Checks

Primary experiment. Using the MIMIC-III antibiotic cohort with a patient-level train/test split, we report $G$ and $G_K$ curves with bootstrap intervals over patients and over model refits.

Observational validity check. Following the deviation analysis in Deep Reinforcement Learning for Sepsis Treatment, we ask whether held-out patients whose actual course matched their assigned stratum's rule show lower reinitiation than those who deviated. The test uses propensity adjustment for the time-varying confounders already represented in the belief.

Calibration check. The fitted model must predict reinitiation rates per belief decile within held-out confidence bands. If it fails, all downstream gains are discounted.

Synthetic check. On synthetic POMDPs, the estimated $G$ should track the true $G$ as patient-belief dispersion grows, and greedy compression should match the exhaustive optimum on small instances.

Secondary testbed. If the Athena screening cohort used by DPSCREEN is accessible, we will apply the same decomposition there. The aim is to ask how much of DPSCREEN's reported 28–68% reduction comes from baseline risk versus interim self-reports. That paper motivates this question but does not answer it.

Decisive outcome. The idea succeeds if three things hold together: $G$ is clearly positive, $G_K$ falls steeply by $K\le 3$, and the α-vector strata beat matched risk-quantile strata out of sample. It fails informatively if risk-quantile strata match it. In that case the result is a principled certificate that a simple stratified guideline is near-optimal for this decision.