Spending Expert Counterfactual Labels Only Where the Data Cannot Speak

After cardiac arrest, many comatose patients die because doctors withdraw life support, not because their brains could not recover. For these patients the recorded outcome says nothing about recovery potential. A prediction model trained on those outcomes learns to reproduce past withdrawal decisions. One recent study fixed this by asking three to five neurocritical care experts per patient to judge whether each withdrawn patient would have woken up. The fix worked, but it is expensive, and it used only a single snapshot of admission data. We propose a hybrid. For withdrawn patients who looked like patients who kept receiving care, statistical reweighting stands in for their missing outcome. Expert judgment is requested only for withdrawn patients whose brain-monitoring trajectory has no comparable patients who stayed on care.

The underlying observation is causal. A withdrawal decision is made at a specific hour, on the basis of an evolving record. If the withdrawal hour and the electroencephalogram (EEG, the continuous recording of brain electrical activity) leading up to it are available, then two kinds of withdrawn patient can be told apart. For the first kind, the data can reconstruct the missing outcome by reweighting similar patients who stayed on full care. For the second kind, withdrawal was nearly certain given what clinicians saw, so no comparable patients exist and only expert judgment can fill the gap. A dynamic model can draw this line. A static model cannot. We expect this to cut the expert-review workload several-fold at matched accuracy on withdrawn patients. We also expect it to soften the calibration penalty that expert alignment currently imposes on patients whose outcomes were observed.

Core Claim

Existing work splits along two axes, and neither combines them. Shen et al. studied 2,497 post-arrest patients. Of these, 1,429 were "uncertain" because care was withdrawn or a non-neurological death intervened. They trained a network with an extra term that pulls predictions toward expert counterfactual probability intervals. At equal accuracy on patients with observed outcomes (area under the ROC curve, AUROC, about 0.88), this cut mean absolute error on uncertain cases from 0.41 to 0.05. This paper motivates the proposal and supplies the alignment loss we reuse.

It also constrains the proposal in three ways. The model is static, which means the time of withdrawal and the preceding trajectory play no role. Every uncertain case gets multi-expert review. Alignment worsens the Brier score, a measure of probability calibration, on observed cases.

The MLHC 2023 competing-risks study supplies the dynamic backbone. On 922 patients it updates forecasts hourly and treats awakening, withdrawal and other death as three competing events. It then treats withdrawal as a terminal endpoint rather than asking what would have happened without it.

Our claim is specific. Suppose the propensity of withdrawal is modeled hour by hour from the evolving record. Then the uncertain cases split into two groups:

  • High-overlap cases. Continued-care patients with similar histories exist. Inverse-probability-of-censoring weighting (IPCW) recovers these cases. IPCW treats withdrawal as censoring and up-weights comparable continued-care patients.
  • Low-overlap cases. Withdrawal was nearly certain, so the positivity assumption fails and no comparators exist. Only expert annotation adds information here.

Directing the expert budget to the low-overlap group should match full-annotation performance with a fraction of the reviews.

Research Program

Model. At landmark hours $t$ (hourly over the first 12 to 72 hours after return of circulation), each patient still under care has a history $H_{it}$. This history is the static admission features plus hourly EEG summaries. We deliberately keep these summaries simple, because Dionisio et al. and the competing-risks study both found that the most recent hour carries most of the signal.

We fit two components on this history:

  • Withdrawal hazard. $\pi(H_{it})$ is the probability that care is withdrawn at hour $t$, given survival and continued care up to $t$.
  • Outcome model. $f(H_{it})$ predicts good neurological recovery under continued care, measured by the Cerebral Performance Category, a five-level scale of post-arrest neurological function.

For a patient withdrawn at hour $\tau_i$, define the overlap score

$$o_i = 1 - \pi(H_{i\tau_i}),$$

the estimated chance that a patient with this exact history would have stayed on care.

Training objective. The outcome model is trained with

$$\mathcal{L} = \sum_{i \in \text{observed}} w_i \,\ell\big(f(H_i), y_i\big) + \lambda \sum_{j \in \mathcal{A}} d\big(f(H_{j\tau_j}), I_j\big).$$

The symbols are:

  • $w_i = \prod_{s<T_i} (1-\pi(H_{is}))^{-1}$ is the stabilized, truncated inverse probability of remaining on care through patient $i$'s follow-up time $T_i$.
  • $\ell$ is the log loss against the observed outcome $y_i$.
  • $\mathcal{A}$ is the set of annotated uncertain cases.
  • $I_j$ is the expert consensus interval for case $j$.
  • $d$ is the interval-distance alignment term of Shen et al., weighted by $\lambda$.

Annotation policy. Cases enter $\mathcal{A}$ in order of lowest overlap $o_i$. Ties are broken by disagreement between the reweighted model's prediction and a bootstrap ensemble of that model.

Secondary hypothesis on calibration. Alignment now constrains only the region where observed data are silent. It should therefore pull less against calibration on observed cases than full alignment does. This is our proposed answer to the calibration trade-off Shen et al. report, and it is directly testable.

Data. The counterfactual component needs withdrawal timing and expert intervals. Both exist only in the single-center Pittsburgh-style cohorts behind the two papers above, so this work requires access to that data. We use the multi-center I-CARE database only for its outcome labels, as a check that the dynamic outcome head transfers across hospitals. It contains 1,020 patients from seven hospitals, used in the PhysioNet 2023 Challenge (Reyna et al.). The file does not document withdrawal information for I-CARE, so we make no counterfactual claim there.

Baselines

Naive reference. A Fine and Gray competing-risks model on the 43 static features plus the latest hour of EEG summaries, trained on observed outcomes with withdrawal as a competing event, following the competing-risks study. That paper reports AUROC 0.923 at 12 hours for 24-hour-ahead prediction, matching Dynamic-DeepHit. This is the strongest baseline on observed cases and embodies the "less is more" finding. We refit it on our splits with the same features and tuning budget. Beating it means non-inferior AUROC on certain cases together with substantially lower error on uncertain cases.

Strongest existing counterfactual method. The static aligned network of Shen et al. with every uncertain case annotated. It reports MAE 0.05 on uncertain cases at about 0.88 AUROC on certain cases. We rerun it on identical splits and give it the same hyperparameter search over $\omega_{bad}$ and $\lambda_{align}$. Beating it does not require lower MAE. It means reaching MAE within a pre-registered margin of 0.02 while using at most 30% of its annotations, with equal or better Brier score on certain cases.

Budget-matched controls. The same dynamic model with the same annotation budget, but with cases chosen at random or by standard uncertainty sampling instead of by overlap.

Ablations. Two ablations isolate the key ingredient. First, IPCW alone with no annotations, which tests whether expert labels are needed at all. Second, the full method with the withdrawal propensity computed from static admission features only. This tests whether the trajectory is what makes the overlap split informative. No reported result was found on Lacuna for XGBoost or TabPFN on the counterfactual cohort, so we report these ourselves as additional tabular references.

Why not just…?

Why not just use IPCW and skip experts entirely? This is the most serious threat, because if withdrawal depends only on the recorded trajectory, reweighting is sufficient. It fails for two reasons. Clinicians act on information not in the EEG summaries, such as pupils, the bedside exam and imaging. And for the worst trajectories withdrawal is near-universal, so the truncated weights explode or become uninformative. The IPCW-only ablation measures this directly. If it matches the hybrid on held-out expert intervals, we will report that expert labels are unnecessary given trajectories, which is itself a useful result.

Why not just annotate cases by random or uncertainty sampling? Model uncertainty is not the same as unidentifiability. A reweighted model can be confident and wrong exactly where positivity fails. The budget-matched controls settle this by plotting error against number of annotated cases for each selection rule.

Why not just train Fine and Gray on the latest hour and drop uncertain cases? That model achieves strong AUROC, but only on patients whose outcomes were observed. Shen et al. show that observed-only models err by 0.41 on uncertain cases, which are 57% of the cohort. Our naive baseline reproduces this failure on our splits.

Why not just use self-supervised EEG embeddings or a foundation model? Banville et al. show self-supervised EEG features help when labels are scarce. EEG-Bench shows that LDA with handcrafted features beats foundation models on mild traumatic brain injury (0.813 accuracy), and that only LaBraM tolerates changed electrode layouts. Better features do not change which label is missing. We test self-supervised embeddings only as an optional input swap, not as a substitute for the method.

Proposed Experiments

The first decisive check is semi-synthetic, because it is the only setting with known counterfactual truth:

  • Take the certain cases, whose outcomes are known under full care.
  • Impose a simulated withdrawal policy driven by trajectory features plus a hidden variable, with strength tuned to reproduce the observed withdrawal rate.
  • Hide the outcomes of the "withdrawn" patients.
  • Sweep the hidden variable's weight and the policy's determinism.

Prediction: IPCW alone succeeds when the policy is mild and has no hidden driver. It fails under near-deterministic withdrawal, where overlap-targeted annotation recovers most of the gap. In this setting, simulated experts return noisy versions of the true label.

The second check uses the real cohort. Existing expert intervals are subsampled to emulate budgets from 5% to 100%. A held-out set of annotated uncertain cases serves as the reference. We plot MAE to the consensus interval against budget for each selection rule.

We also report the following for every method:

  • AUROC, Brier score and true positive rate at false positive rate ≤ 0.05 on certain cases.
  • The same metrics at each landmark hour.

The win condition is Shen-level MAE at 30% budget with no Brier degradation relative to the observed-only model.

The third check applies the dynamic outcome head to I-CARE under leave-one-hospital-out evaluation. It will be compared only against the Challenge's observed-outcome scoring. For reference, the winning Challenge entry reached a true positive rate of 0.792 on hidden hospitals, while Jiang et al. fell from 0.418 to 0.238. The goal is to show the backbone is not a single-center artifact, not to claim counterfactual validity there.