When a clinician chooses between several therapies, the decision depends less on the patient's risk and more on whether the patient's physiology has crossed a point where one option overtakes another. Evidence synthesis already has a language for comparing many treatments under uncertainty. The cumulative ranking curve gives the probability that a treatment is among the top $k$, and its area, SUCRA (surface under the cumulative ranking curve), condenses that curve into a single 0–100% score. That language, however, works only at the population level. Machine learning offers the opposite. It is good at finding thresholds, meaning biomarker values where an outcome saturates or reverses, but it finds them on risk models that never compare treatments. This note proposes to estimate patient-conditional cumulative ranking curves over competing ICU therapies and to locate rank-flip thresholds: the biomarker values at which one therapy's probability of outranking another crosses one half. Uncertainty bands would include hospital-to-hospital variation, so that a flip found at one site is not reported with confidence it does not deserve at another.
The practical stakes come from a common pattern. A tree ensemble predicts mortality, a SHAP dependence plot shows an inflection in lactate or blood pressure, and the inflection is read as a place where management should change. That plot describes prognosis, how risk varies with the biomarker. It does not describe effect modification, how the relative benefit of treatments varies with it. The two thresholds need not coincide. The central empirical claim is that thresholds read from outcome-model SHAP plots systematically misplace treatment-relevant thresholds. Direct estimation of rank flips, with predictive (new-site) uncertainty, recovers them and transfers across hospitals better.
Core Claim and Definitions
For a patient with covariates $x$ and $K$ candidate therapies with potential-outcome means $\mu_1(x),\dots,\mu_K(x)$, the patient-level SUCRA of therapy $a$ is
$$\mathrm{SUCRA}_a(x)=\frac{1}{K-1}\sum_{k=1}^{K-1}P\big(\mathrm{rank}_a\le k \mid x\big),$$
where the probability is taken over posterior uncertainty in the $\mu$'s. The predictive version adds a draw of an unseen hospital's random effect. This is the patient-level analogue of the predictive P-score of Predictive P-score for treatment ranking in Bayesian network meta-analysis. That paper adds between-study variance to trial-level rankings and shows confidence shrinking toward 0.5, for example group counseling falling from a Bayesian P-score of 0.879 to a predictive 0.746 in the smoking network. We keep the same logic but treat hospitals as the "studies" and condition on patient covariates. The rank-flip threshold for therapies $a,b$ along biomarker $j$ is
$$t^{*}_{ab,j}(x)=\{\,t:\;P\big(\mu_a(x_{j\leftarrow t})>\mu_b(x_{j\leftarrow t})\big)=\tfrac12\,\},$$
where $x_{j\leftarrow t}$ is the patient's record with biomarker $j$ moved to value $t$. Each threshold is reported with a credible interval for $t$. If the curve never crosses one half with sufficient sharpness, the method reports that no flip was found.
The claim that is new relative to prior work is narrow and testable. Treatment-relevant thresholds are crossing points of conditional ranking probabilities. They can be estimated and attributed directly. Doing so beats the two things practitioners actually do now: reading thresholds off prognostic SHAP plots, and sorting point estimates from treatment-effect models.
How This Relates to Prior Work
Three groups of papers motivate the gap. First, population-level SUCRA practice, represented by network meta-analyses such as Comparative efficacy and safety of prostacyclin therapies for pulmonary arterial hypertension and the predictive P-score paper above, ranks treatments well but cannot say how a ranking changes for a given patient. Second, threshold detection with gradient-boosted trees and SHAP, as in Data-Driven Spatial Optimization of Elderly Care Facilities, which reports an accessibility saturation point at 6.0, finds inflections in one outcome and never in a comparison between options. Third, clinical SHAP work such as Explainable machine learning for predicting ICU mortality in myocardial infarction patients using pseudo-dynamic data shows that acute biomarkers dominate risk near deterioration. It is purely prognostic, and it is exactly the practice whose thresholds we expect to be misread as treatment thresholds.
Two lines of work supply method. Rank-Learner: Orthogonal Ranking of Treatment Effects builds Neyman-orthogonal pairwise ranking pseudo-labels with cross-fitting, which makes rankings robust to errors in the propensity and outcome models. Metalearners for Ranking Treatment Effects shows that ranking metrics, not mean squared error, track policy value. Both rank patients for a single treatment. We reuse the orthogonal pseudo-outcome idea for pairs of arms within a patient and add rank distributions over many arms. Delta-XAI supplies the attribution technique. Its SWING method integrates along a patient's real historical trajectory instead of an unphysical zero baseline, and it enforces completeness, meaning the attributions sum to the change in the explained quantity. We apply SWING to the pairwise ranking margin rather than to scalar risk.
One paper constrains the design. Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap argues that post-hoc explanations can be unfaithful and favors transparent ante-hoc models. We take this seriously in two ways. Our primary output, the flip threshold with its band, is a property of the estimated ranking rather than an approximation of it. And attribution is used only to explain movement across a flip, where completeness can be checked numerically. The translation-gap literature sets the evaluation regime. Deploying clinical machine learning? Consider the following.. reports that 72% of FDA-approved AI devices lack public multi-site evaluation, so every real-data claim here is judged on transfer from eICU to MIMIC-IV.
Research Program
The estimator has three parts. The first is a multi-arm outcome model with hospital random effects, fit on eICU with cross-fitted propensity and outcome models. Its posterior is approximated by a Bayesian bootstrap over patients nested within hospitals, plus a Gaussian random effect whose variance plays the role of the network meta-analysis heterogeneity term. Orthogonal pairwise pseudo-outcomes in the style of Rank-Learner correct each arm pair. The second part produces patient-level rank histograms, cumulative ranking curves, and SUCRA, in both "same hospital" and "new hospital" versions. The third part scans each biomarker along clinically realistic paths to find flip thresholds, then attributes the change in predictive SUCRA difference across the flip with SWING. Scanning a biomarker alone through implausible combinations would create out-of-distribution artifacts, so paths are taken from observed trajectories or matched neighbors.
The application is early sepsis resuscitation. The therapies are a small number of fluid and vasopressor strategies coarsened from the 25-bin action space of ICU-Sepsis: A Benchmark MDP Built from Real Medical Data. The coarsening is chosen so that every strategy has adequate overlap, because confounding by indication is severe here. Real-data flips are therefore treated as hypotheses. Correctness is established on data with known ground truth.
Baselines
The naive reference point is population SUCRA, which gives every patient the same ranking computed with no covariates. This is how network meta-analysis output enters guidelines today. Any patient-level method must beat it on ranking accuracy, or it has added nothing. A secondary sanity check is that our predictive SUCRA, with covariates removed and applied to the smoking-cessation network, reproduces the reported trial-level predictive P-score of 0.746 for group counseling. That figure is reported in the predictive P-score paper.
The strongest competitor for finding thresholds is current practice. It fits an XGBoost outcome model on the same eICU features in the manner of XMI-ICU, then reads inflection points from SHAP dependence plots, either including treatment as a feature or fitting one model per arm. For context, XMI-ICU reports a 6-hour mortality AUROC of 92.0 on eICU and roughly 75.1–80.0 on external MIMIC-IV. Our comparison is about threshold location, not AUROC. It gets the same features, the same hyperparameter search budget, and the same cross-fitting folds. It is beaten if our flip thresholds are closer to planted truth in mean absolute error and their intervals achieve nominal coverage where its point readings do not.
The strongest competitor for the ranking itself is a multi-arm doubly robust learner. It is extended with a bootstrap to produce rank distributions and given an equal tuning budget, with Rank-Learner's pairwise objective applied to each arm pair as a second variant. Rank-Learner reports higher AUTOC than T- and DR-learners, but no reported numeric result was found on Lacuna for the semi-synthetic MIMIC-III setting, so we rerun its released code rather than quote a number. Beating these means higher within-patient Kendall's $\tau$ against true arm orderings and better calibration of predictive rank probabilities on a held-out hospital.
For policy context on ICU-Sepsis, the reported expected survival is 0.88 for the optimal policy, 0.86 for DQN and PPO, and 0.78 for both the clinician and random policies. A SUCRA-maximizing policy should approach 0.86–0.88 while also supplying rank uncertainty.
Three ablations isolate the ingredients. Removing the hospital random effect tests whether predictive SUCRA matters for transfer. Replacing SWING's trajectory path with a zero baseline tests whether the attribution remains complete and stable. Replacing SUCRA with $P(\mathrm{rank}=1)$ tests whether the full curve changes choices in high-variance patients.
Why not just…?
Why not just fit a multi-arm treatment-effect model, bootstrap it, and sort? This is the real threat, because our estimator is close to it. If a bootstrapped DR-learner yields the same flip locations, the same coverage, and the same cross-hospital calibration, then the contribution reduces to the flip-reading procedure and the heterogeneity term. The ablations are built to show this. Removing the random effect and the orthogonal pairwise correction turns our method into this baseline. If neither removal costs accuracy on held-out hospitals, we will report that the simpler pipeline suffices. The evidence from Metalearners for Ranking Treatment Effects, that pointwise error barely tracks ranking utility, suggests sorting point estimates will lose on within-patient ordering, but it does not guarantee this.
Why not just read thresholds off SHAP plots of a good mortality model? Those plots locate where risk bends, not where the best therapy changes. The planted-threshold experiment separates the two: prognostic inflections and rank-flip points are placed at different biomarker values. If the SHAP-reading baseline tracks the flips anyway, the core claim fails.
Why not just train a reinforcement-learning policy? DQN reaches 0.86 on ICU-Sepsis, but it outputs a single action with no statement of how close the runner-up is. The ICU-Sepsis authors also document a sparsity trap, where policies exploit rare transitions and report unrealistic survival. Rank histograms and flip intervals expose exactly when two actions are not distinguishable. We test whether the margins near our flips coincide with states where the reinforcement-learning policy's choices are unstable across seeds.
Why not just report the treatment most likely to be best? That rule selects volatile options with high chances of being both best and worst. The $P(\mathrm{rank}=1)$ ablation measures how often it changes the chosen arm, and what that costs in true value, on ICU-Sepsis.
Proposed Experiments and Decisive Checks
The first experiment is semi-synthetic, with real covariates and planted thresholds. Outcomes are simulated on real eICU and MIMIC-IV covariates in the spirit of Rank-Learner's semi-synthetic MIMIC-III design. Each arm's effect is planted to saturate or reverse at known lactate, mean arterial pressure, or creatinine values. Prognostic inflections are placed elsewhere, and hospital-level effect variation of controlled size is added. The decisive check is threshold-location error and interval coverage versus the SHAP-reading baseline, swept over heterogeneity levels. As heterogeneity grows, predictive intervals should widen and keep coverage while same-hospital intervals undercover. This mirrors the shrinkage the predictive P-score paper reports, and it is the reason predictive uncertainty is included.
The second experiment uses ICU-Sepsis as an oracle. Because the benchmark specifies its transition model, true action values per state are computable. We fit our estimator to trajectories sampled from the clinician policy and measure within-state Kendall's $\tau$, calibration of rank probabilities, and the value of the SUCRA-maximizing policy against the reported 0.78, 0.86, and 0.88 anchors.
The third experiment is real transfer. We train on eICU and estimate flips and predictive SUCRA, then check on MIMIC-IV whether flips replicate within their intervals. We also check whether predictive rank probabilities are calibrated, using doubly robust pseudo-outcomes as targets, and whether the zero-baseline ablation breaks completeness of the attribution. Real-data flips that replicate are reported as hypotheses for prospective study, not recommendations.
The proposal fails cleanly if either of two things happens. One is that the SHAP-reading baseline locates planted flips as well as our method does. The other is that a bootstrapped multi-arm learner without the heterogeneity term matches our coverage on held-out hospitals. Either outcome is informative, and both are reported.
References
- Predictive P-score for treatment ranking in Bayesian network meta-analysis
- Comparative efficacy and safety of prostacyclin therapies for pulmonary arterial hypertension: a systematic review and network meta-analysis
- Data-Driven Spatial Optimization of Elderly Care Facilities: A Study on Nonlinear Threshold Effects Based on XGBoost and SHAP - A Case Study of Xi'an, China
- Explainable machine learning for predicting ICU mortality in myocardial infarction patients using pseudo-dynamic data
- Rank-Learner: Orthogonal Ranking of Treatment Effects
- Metalearners for Ranking Treatment Effects
- Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoring
- Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap
- Deploying clinical machine learning? Consider the following..
- ICU-Sepsis: A Benchmark MDP Built from Real Medical Data