Hospital-Invariant Harm Phenotypes for NSAID Kidney Injury in the ICU

Painkillers such as ibuprofen and ketorolac (non-steroidal anti-inflammatory drugs, NSAIDs) are widely suspected of injuring the kidneys of critically ill patients. Large observational studies nonetheless report that, on average, they look harmless or even protective. The usual explanation is that healthier patients are the ones who receive them. An average can also hide a minority who are genuinely harmed. Machine learning offers to find that minority by clustering patients into data-driven "phenotypes". In critical care, however, such phenotypes have a poor replication record: a subgroup that seems to respond differently in one trial often vanishes in the next.

We propose to treat disagreement between hospitals as the test that separates real effect-modifying phenotypes from artifacts. Using the eICU database, which spans more than 200 US hospitals, we will learn patient groups whose NSAID-versus-acetaminophen kidney-injury effect differs between groups and stays the same across hospitals. We will then ask whether those groups predict effect differences in hospitals and whole databases never seen during training, namely MIMIC-IV, AUMCdb and HiRID. Held-out-hospital replication is the yardstick. The proposal is equally a benchmark and a method. Simpler stratifications, notably baseline-risk strata, compete on the same yardstick, and finding that they win would itself be a useful answer.

Core Claim

Two failure modes recur in the file. The first is the "underfitting trap" described by Smit, Jonkman and Krijthe, who argue that critical care sub-phenotyping clusters on baseline covariates alone. Because it ignores treatment and outcome, well-separated clusters need not differ in treatment response, and the hyper-inflammatory ARDS benefit from high PEEP (positive end-expiratory pressure) seen in ALVEOLI did not replicate in LOVS. The second failure mode comes from pooling across sites. YAIB shows that site differences in cohort and label definitions move results more than model choice, and that models degrade across databases.

Our claim is that both problems can be addressed by one design choice. Phenotypes should be fit to estimated treatment effects rather than to covariates. They should then be explicitly penalized when their effect contrast varies across hospitals. Finally, they should be judged only on hospitals held out from discovery. The new element is using many-hospital heterogeneity as a regularizer and as an evaluation protocol for subgroup safety claims, applied to a concrete, clinically contested drug–outcome pair.

Research Program

The first deliverable is a harmonized analgesic exposure layer. YAIB explicitly does not harmonize medications, and PyHealth maps drug codes but provides no exposure-timing logic. We will build an extension to the YAIB/ricu cohort pipeline that extracts first NSAID administration and acetaminophen administration, with route and dose, from the medication tables of eICU, MIMIC-IV, AUMCdb and HiRID.

On that layer we emulate a target trial, meaning an observational analysis set up to mimic a hypothetical randomized trial's eligibility, start time and follow-up. Eligible patients are adults who receive a first non-opioid analgesic within 48 hours of ICU admission. The exposed group receives an NSAID. The comparator group receives acetaminophen only.

Using acetaminophen recipients as an active comparator is deliberate. Reese et al. compared NSAID users with non-users across 38 N3C centers. They obtained an implausible AKI odds ratio of 0.67 (95% CI 0.63–0.72) and attributed it to healthy-user bias, with E-values of 1.9 to 3.3. A patient deemed to need an analgesic is a much closer comparator than a patient deemed to need nothing.

The primary outcome is KDIGO creatinine-defined acute kidney injury (AKI) within 72 hours. KDIGO is the standard international staging rule for kidney injury, and we use YAIB's harmonized AKI definition. All covariates come strictly from the window before exposure. This keeps post-exposure quantities out of the adjustment set, such as urine output falling or vasopressors escalating after the dose. It also satisfies the no-mediator assumption that Shi, Veitch and Blei require for invariant causal estimation.

Covariates combine static features with a sequence encoding of the preceding vitals and labs, in the style of Suresh et al.. We also include the pre-exposure medication profile that Sikora et al. showed carries prognostic signal: in their data, two clusters with similar organ support had 9.0% versus 21.9% mortality.

The method has three stages. First, for each patient $i$ we compute a doubly robust pseudo-outcome $\psi_i$. This is a per-patient effect signal built from a cross-fitted propensity model and outcome model, and its average within any group is a consistent estimate of that group's effect if either model is correct. Second, an encoder assigns each patient soft membership in one of $K$ phenotypes, with $K$ small (2 to 5). Third, writing $\hat\tau_{e,k}$ for the effect in phenotype $k$ estimated from patients in hospital $e$, $\bar\tau_k$ for its pooled value, $\bar\tau$ for the overall effect, and $\pi_k$ for phenotype prevalence, we minimize

$$ \mathcal{L} = -\sum_k \pi_k(\bar\tau_k-\bar\tau)^2 + \lambda\sum_k\sum_e w_{e,k}(\hat\tau_{e,k}-\bar\tau_k)^2 . $$

The first term rewards phenotypes whose effects differ. The second term, weighted by $\lambda$ and by precision weights $w_{e,k}$, penalizes phenotypes whose effect shifts from hospital to hospital. The learned assignment is then distilled into a shallow decision tree, so that each phenotype can be read as a bedside rule.

This builds on Counterfactual Phenotyping with Censored Time-to-Events, which separates baseline risk from treatment-response phenotypes in randomized trials (ALLHAT, ACCORD). We differ in three ways: our data are observational, they come from many sites, and our training objective addresses cross-site replication. Our approach also differs from Wu et al., whose self-organizing-map subgroups replicated across two trials but whose boundaries were drawn by eye. Here the boundaries come from the objective and are tested on hundreds of held-out hospitals.

The program also includes an operational-artifact check. Care Phenotypes In Critical Care shows that bedside care delivery varies systematically with patient characteristics. A phenotype could therefore track how a hospital delivers care rather than a patient's biological susceptibility. For every learned phenotype we will report how well hospital identity predicts membership compared with how well physiology predicts it.

Baselines

The naive reference is pooled 1:1 propensity score matching with a single average effect, modeled on Reese et al.. Their reported AKI result is OR 0.67 (95% CI 0.63–0.72) with a non-user comparator. That figure comes from N3C, not from our active-comparator ICU cohort, so it frames the problem rather than serving as a target. We rerun the method on our cohort to obtain the "no subgroups" reference.

The covariate-only phenotyping baseline reproduces the underfitting trap. We cluster the same pre-exposure features, using a sequence autoencoder with a Gaussian mixture model as in Suresh et al., plus the medication-profile clustering of Sikora et al.. We then estimate effects within the resulting clusters.

Risk-based stratification is the baseline Smit et al. recommend. It is the strongest principled competitor, because effect heterogeneity often tracks baseline risk. We fit a gradient-boosted model of baseline 72-hour AKI risk on the comparator arm and estimate effects within risk quintiles. YAIB reports mortality AUROC of 0.84–0.87 for its benchmark models across databases. For a pre-exposure AKI risk model, no reported result was found on Lacuna, so we will report our own.

The direct effect-modeling baseline fits a causal forest or a doubly robust learner of individual effects to the same pseudo-outcomes. We then group patients by quantiles of predicted effect, with hospital-clustered sample splitting. No reported result was found on Lacuna for any such estimator on ICU analgesic outcomes.

The key ablation sets $\lambda=0$. This keeps the outcome-aware objective but removes the invariance penalty, isolating the contribution of the cross-hospital term.

All methods receive the same patients, the same pre-exposure features, the same cross-fitted nuisance models and the same hospital-level discovery/replication splits. Each gets the same Bayesian hyperparameter search budget as in YAIB, with the number of groups matched at $K$. Beating a baseline means a larger replicated between-group effect contrast in held-out hospitals, with a calibration slope closer to 1, across repeated splits. Better in-sample separation does not count.

Why not just…?

Why not just stratify by baseline AKI risk, as Smit et al. advise? This is a real threat, and the experiments are built so it can win. Risk strata are simple, they replicate well because a risk model is transportable, and NSAID harm may simply grow with renal vulnerability. The proposal's added value exists only if part of the replicable heterogeneity is qualitative. Sikora et al.'s clusters with similar organ support but different mortality suggest that patients can look alike on one axis and differ on another. Our decisive comparison measures how much held-out effect variation remains after conditioning on risk quintile. If that residual is null, we will report that risk stratification suffices, and the harmonized benchmark remains the contribution.

Why not just fit a causal forest and threshold its predictions? Direct effect models already use treatment and outcome, so they escape the underfitting trap. Nothing in them, however, discourages fitting effect modifiers that hold in one hospital only. Shi et al. show, in a different role (choosing which variables to adjust for), that enforcing invariance across environments removes spurious structure that "adjust all" methods absorb. Whether the same holds for subgroup effects is exactly what the $\lambda=0$ ablation and the causal-forest baseline test on held-out hospitals.

Why not just meta-analyze per-hospital estimates with random effects? That approach quantifies how much the average effect varies across sites. It does not say which patients drive the variation. Practice-pattern differences can also masquerade as patient differences. We report the random-effects heterogeneity as a descriptive companion, and we use the hospital-predictability check to flag phenotypes that are site artifacts.

Proposed Experiments

The primary experiment repeatedly splits eICU hospitals into discovery and replication halves, using at least 50 splits stratified by hospital size and region. Every method learns its groups on discovery hospitals. Group effects are then re-estimated, with fresh nuisance models, on the replication hospitals. We report the replicated between-group contrast, the sign-agreement rate and the calibration slope between discovered and replicated group effects.

External transport applies the frozen eICU phenotypes to MIMIC-IV, and to AUMCdb and HiRID where enough patients receive NSAIDs. A pre-registered feasibility check counts exposed patients and examines propensity overlap per database. Any database failing it is reported but excluded from inference.

Three decisive checks complete the program. In a placebo test, treatment is permuted within hospitals; every pipeline should then find no replicating subgroups, and the rate at which each one still does estimates its false-discovery rate. In a semi-synthetic test, outcomes are simulated on real eICU covariates and treatments, with one planted biological effect modifier and one planted hospital-specific spurious modifier. The test asks whether the invariance penalty recovers the first while rejecting the second, where covariate clustering and the $\lambda=0$ variant do not. The third check is clinical. Any surviving phenotype must be described as a shallow rule, for example a combination of pre-exposure creatinine trajectory, vasopressor dependence and concurrent nephrotoxic drugs, and it must pass the hospital-predictability check before we call it a harm phenotype.