Borrowing Evidence to Evaluate Models for Rare ICU Conditions

When a hospital has only fifty patients with a rare condition, two jobs are hard. Training a good model is one. Knowing whether the model is any good is the other, and it gets far less attention. The field has made progress on the first: methods now borrow patients from clinically similar, more common conditions to train better predictors. The second job has been left to ordinary cross-validation, which means repeatedly holding out part of the data, training on the rest, and averaging the scores. With fifty patients and a handful of positive outcomes, each held-out slice contains a few events at most. The resulting score is so noisy that it cannot reliably tell a good transfer method from a harmful one. The consequence is that the decisions that matter most go unchecked: which source conditions to borrow from, how strongly, and whether to borrow at all.

We propose to evaluate models for rare conditions by borrowing evidence the same way training already borrows data. For each candidate model, we combine its noisy cross-validated score on the rare target cohort with its scores on clinically similar source conditions. The weight on each part is set by how noisy the target score is and by how much performance really varies between conditions. This is empirical-Bayes shrinkage, the small-area estimation idea long used in survey statistics, applied to model selection for rare clinical cohorts. To test it, we build a benchmark from MIMIC-III and eICU where the true performance of every candidate model is known. We create "pseudo-rare" cohorts by subsampling common conditions down to 30–200 patients and keep the thousands of remaining patients as a ground-truth test set. We then measure which evaluation procedure picks the model that actually works best.

Core Claim

In rare-condition cohorts of fewer than roughly 200 patients, the main source of error is often the choice among training strategies, not the training itself. A similarity-weighted shrinkage estimate of each candidate's performance should cut selection regret substantially compared with target-only cross-validation, repeated or nested. Regret here is the true-performance gap between the chosen model and the best available one. The gain should be largest at the smallest cohort sizes and should disappear as the target cohort grows, when the target score becomes precise enough on its own. Two things would make this new. The first is treating evaluation of transfer methods as a borrowing problem in its own right. The second is a benchmark that makes regret measurable, which no study in this area currently offers.

Why This Gap Is Real

Three kinds of prior work motivate the proposal.

The first is transfer training for rare conditions. Bridging data gaps of rare conditions in ICU, called KnowRare, builds a condition knowledge graph from diagnosis co-occurrence, lab distributions and drug overlap. It uses the graph to choose similar source conditions and adapts representations adversarially. It also documents a "data-volume paradox": pooling all conditions hurts rare ones. KnowRare is evaluated on fixed splits, so its own choice of how many sources to use was never subjected to an evaluation designed for tiny cohorts. We reuse its similarity graph but point it at evaluation instead of training.

The second is sample-efficient transfer. Boosting Transfer Learning with Survival Data from Heterogeneous Domains reweights source patients by similarity to the target. Transfer Causal Learning penalizes sparse differences between source and target parameters. It shows that on a target of 58 treated patients, target-only and naively merged estimators both reverse the sign of a known treatment effect. That result is the clearest evidence that picking the wrong strategy in small cohorts produces qualitatively wrong conclusions, not just slightly worse ones. Our shrinkage estimator is the evaluation-side counterpart of the sparse-difference idea: target performance is assumed close to, but not equal to, similar-source performance.

The third is theory that constrains the design. Cross validation for rare events shows that cross-validation error on a region of probability mass $\nu_\alpha$ shrinks only as $O(\sqrt{\nu_g/(n\nu_\alpha)})$. This follows from concentration arguments that effectively leave only $n\nu_\alpha$ usable samples. With tiny $n$ and rare outcomes, the bound is loose enough to be uninformative. The same paper's guarantees require bounded losses, so we score with the Brier score and precision-recall area, never unregularized log-loss. Counterfactual Cross-Validation shows that the choice of validation procedure changes which model gets selected. However, it was studied on IHDP with 747 units and fits its own internal regression. That regression is exactly what destabilizes in the regime flagged by Bria et al., who note that flexible models memorize artifacts below about 100 patients. We do not propose CF-CV as a fix. We treat it as one input that shrinkage could stabilize, which we test in a secondary causal arm.

Research Program

The estimator is simple, which is the point. Let $m$ be a candidate model, meaning a full training strategy such as target-only logistic regression, pooled training, KnowRare with $k$ sources, or similarity-reweighted boosting. Let $\hat R_t(m)$ be its repeated cross-validated score on the target cohort $t$, with sampling variance $\sigma_t^2(m)$ estimated by bootstrap. Let $\hat R_s(m)$ be its score when the same strategy is applied with condition $s$ playing the target role. Let $\pi_{ts}$ be normalized similarity weights from the KnowRare condition graph. The shrinkage estimate is

$$\tilde R_t(m) = w_m \hat R_t(m) + (1-w_m)\sum_s \pi_{ts}\hat R_s(m), \qquad w_m = \frac{\tau^2}{\tau^2+\sigma_t^2(m)},$$

where $\tau^2$ is the between-condition variance of true performance around the similarity-weighted mean. We estimate $\tau^2$ by method of moments across many source conditions where scores are precise. We then select the $m$ with the best $\tilde R_t(m)$. When the target is large, $\sigma_t^2$ is small and the estimate reduces to ordinary cross-validation. When the target is tiny, it leans on similar conditions, but only as far as conditions have been shown to agree.

The build has three parts. First, a candidate pool of 8–12 training strategies spanning the choices a practitioner actually faces. Second, the pseudo-rare benchmark. We take ICU conditions with at least 2,000 stays in MIMIC-III or eICU and repeatedly subsample training-and-validation cohorts of 30, 50, 100 and 200 stays. Each candidate is scored on the held-out remainder of that condition, which gives its true performance. Third, a check on genuinely rare conditions, where no large held-out pool exists. There, eICU's multi-hospital structure lets us train on some hospitals and score on patients from unseen hospitals. This is noisier but external. The check addresses the main weakness of pseudo-rare cohorts: subsampling a common condition does not reproduce the heterogeneity of a truly rare one.

The tasks are the ICU prediction tasks used by KnowRare (90-day and ICU mortality, 30-day readmission) plus in-hospital mortality from Multitask Learning and Benchmarking with Clinical Time Series Data. The primary metric is selection regret in true precision-recall area. Secondary metrics are the error of the performance estimate itself and the rank correlation between estimated and true performance across candidates.

A secondary causal arm asks whether the same idea stabilizes treatment-effect model selection. We graft IHDP-style simulated outcome surfaces onto real covariates of small rare-condition cohorts, so that true effects are known. We then compare CF-CV on the target alone with CF-CV shrunk toward similar conditions.

Baselines

The naive reference point is a single stratified 5-fold cross-validation on the target cohort, used to pick among the candidates. It is what most small clinical studies do, and it shows how much regret current practice incurs.

The strongest competitor for the evaluation task is repeated nested cross-validation on the target alone. It uses 50 repetitions, with the variance-corrected standard error used to break near-ties in favor of the simpler model, a one-standard-error rule. It is strongest because it squeezes the most variance reduction out of target data without any borrowing assumption, and a reviewer would reasonably expect it to be enough. Every evaluation method gets the same candidate pool, the same trained models and the same compute budget per cohort draw, so only the scoring rule differs. For this baseline we found no reported result on Lacuna for selection regret or estimate error at cohort sizes under 200. Establishing that number is part of the contribution. We count the proposal as beating it if shrinkage lowers median regret by at least a quarter at N=50, with bootstrap confidence intervals over at least 200 cohort draws per condition that exclude zero, and does no worse at N=200.

For the training strategies themselves, the strongest existing method is KnowRare. Its paper reports "improvements of up to 17.0% in AUPRC compared to the best domain adaptation method" across MIMIC-III and eICU (Bridging data gaps). It enters the candidate pool at several source counts, so the question becomes whether each evaluation procedure can tell when KnowRare helps a given small cohort and when it does not. Reference points from the file frame how much is at stake. Logistic regression reaches an AUC-ROC of 0.848 against 0.870 for a channel-wise LSTM on full MIMIC-III mortality (Multitask Learning and Benchmarking). A target-only Cox model scores a C-index of 0.500 against 0.677 for transfer boosting on the MAGGIC Euro cohort (Boosting Transfer Learning). Both suggest the gap between strategies can be large enough that picking wrongly is costly.

Two ablations isolate the key ingredient. The first replaces the knowledge-graph similarity weights $\pi_{ts}$ with uniform weights over all source conditions. This tests whether borrowing evidence only from similar conditions matters or whether any averaging helps. The second fixes $w_m$ at a constant instead of estimating it from $\sigma_t^2$ and $\tau^2$. This tests whether adapting the weight to each model's noise level is needed.

Why not just…?

Why not just run more cross-validation repeats? Repeating cross-validation reduces the noise that comes from how the data are split. It does not reduce the noise that comes from having only a few outcome events in the whole cohort. The rare-events bound above depends on $n\nu_\alpha$, the number of usable samples, not on how many times the data are re-split. Repeated nested cross-validation is nonetheless our strongest baseline, and it could win. If the regret curves at N=50 show it matching shrinkage, we will report that borrowing evidence is unnecessary.

Why not just evaluate on the pooled source conditions, which have plenty of data? That estimate is precise but biased toward conditions that are not the target. The data-volume paradox in KnowRare and the sign reversal under merging in Transfer Causal Learning show that source and target can disagree about which strategy is best. We include source-only evaluation, similarity-weighted with $w_m=0$, as a baseline. The regret it incurs on cohorts where sources mislead is the direct test.

Why not just use leave-hospital-out validation? Holding out whole hospitals addresses site shift. Portable representations for clinical time series shows that validating within a single site underestimates cross-site error. For rare conditions, though, each hospital holds only a few patients, which makes the variance problem worse. We use it where it fits, as the external check on truly rare eICU conditions. We do not use it as a selector.

Decisive Checks

The main result is the regret curve against target cohort size for the three evaluation procedures: target-only, source-only and shrinkage. The idea succeeds if shrinkage stays below both at small N and converges to target-only at large N. Three findings would falsify it. The first is shrinkage losing to repeated nested cross-validation at N=50. The second is the uniform-weight ablation matching the similarity-weighted version, which would mean the knowledge graph adds nothing. The third is pseudo-rare gains failing to reappear on the held-out-hospital check for genuinely rare conditions. The causal arm is judged by the same regret logic against known simulated effects. We will report it as exploratory, because CF-CV's internal regression may be too unstable below 100 patients to benefit from any reweighting.