Two Clocks for Triage: Calibrating Rule-Out Safety on Outcomes and Capacity on Live Scores

Clinical risk models output a continuous score, but hospitals act in tiers: discharge, monitor, or act now. The usual way to set those tiers is to pick cutoffs on a retrospective cohort. A single number then has to carry two very different jobs. The low cutoff decides who is sent home, and missing a dying patient there is catastrophic. The high cutoff decides who consumes scarce resources such as a radiologist within 30 minutes or an ICU bed, and flagging too many people overloads the service. Both jobs are costly when they fail, but in different ways and on different timescales. A missed death shows up in the outcome data weeks later. An overloaded queue shows up the same afternoon. We propose to calibrate the two cutoffs on the two signals that actually reveal their failures. The rule-out cutoff would be set from labelled outcomes, with a finite-sample guarantee on the event rate among discharged patients. The rule-in cutoff would be adjusted continuously from unlabelled live score counts so that it meets a capacity limit. Because the tiers are ordered, moving the rule-in cutoff never touches the rule-out guarantee.

This also sidesteps a problem both directions share. Clinicians cannot reliably write down a numerical cost matrix, as Conformal Risk Control for Ordinal Classification and Learning Decision Thresholds for Risk-Stratification Models from Aggregate Clinician Behavior both note. They can, however, state two things: an acceptable miss rate among people sent home, and how many urgent slots exist per day. We replace the unspecifiable cost ratio with those two quantities and a strict priority order. Safety is never traded for capacity. When the two constraints conflict, the system reports that no safe cutoff exists instead of quietly relaxing safety.

Core Claim

A three-tier triage rule has a rule-out cutoff $t_{out}$ and a rule-in cutoff $t_{in} \ge t_{out}$ on a score $s(x)$. We choose $t_{out}$ as the largest cutoff satisfying a selection-conditioned constraint:

$$\mathbb{E}\big[\,Y\cdot\mathbf{1}\{s(X)<t_{out}\} - \alpha\,\mathbf{1}\{s(X)<t_{out}\}\,\big] \le 0 .$$

Here $Y=1$ marks the adverse outcome (for example, 30-day death), and $\alpha$ is the tolerated event rate among discharged patients. The constraint states that the event rate within the discharge tier is at most $\alpha$. It is calibrated with the finite-sample linear-expectation method of LEC.

The rule-in cutoff $t_{in}$ is updated online so that the long-run fraction of patients with $s(X)\ge t_{in}$ stays at or below a capacity $\kappa$. This update uses only scores, which are available immediately.

The key structural fact is simple. The discharge tier depends only on $t_{out}$. Any rule-in adaptation that keeps $t_{in}\ge t_{out}$, however aggressive and however shifted the deployment distribution, therefore moves patients only between "monitor" and "act." It cannot move them into "discharge." The two calibrations can run on separate clocks without invalidating each other.

What is new relative to prior work has four parts. First, we give a stratum-conditional guarantee rather than a marginal one. Second, we treat capacity as a label-free online quantity rather than a retrospective choice. Third, we give an explicit infeasibility rule when gray-zone capacity also binds. Fourth, we test all of this in the rare-event clinical regime, where it matters most.

Why This Gap Is Real

Two findings in the file motivate the proposal. Deep Learning-Based Approach to Emergency Department Chest Pain Risk Stratification by ECG reports a rule-out cutoff (ECG-Risk < −1.25) below which zero 30-day deaths occurred. That cutoff was chosen by looking at where the deaths stopped. Zero observed events among $n$ discharged patients is compatible with a true rate near $3/n$ by the standard "rule of three" binomial bound, and the post-hoc selection makes even that optimistic. The claim is a hypothesis about safety, not a guarantee.

Triaging mammography with artificial intelligence shows the other failure. A threshold that flagged 14% of cases retrospectively flagged 29% prospectively. This is exactly the drift a fixed retrospective rule-in cutoff cannot absorb, and it was visible on day one without any labels.

Three papers supply methods but stop short of this problem.

  • Conformal Risk Control for Ordinal Classification calibrates a threshold so that severity-weighted expected loss stays below $\alpha$. It controls marginal risk over all patients and outputs label sets. The risk among the discharged, which is the quantity a clinician cares about, can therefore exceed $\alpha$ while the average stays compliant.
  • LEC gives the selection-conditioned calibration we reuse, including joint two-threshold routing. It was built for question answering. Both thresholds are fitted to labelled risk, with no capacity constraint and no notion that one threshold can move without labels.
  • Learning to Offload with Low Regret under Asymmetric Misclassification Costs proves that asymmetric costs plus a costly deferral yield a two-threshold policy. Its guarantees are regret bounds on a cost-weighted objective, not per-tier safety bounds. It also assumes the receiving tier is a perfect, uncapacitated oracle.

Our proposal keeps their two-threshold structure. We change what each threshold is responsible for.

Research Program

Calibration procedure. Given a held-out labelled calibration set, we compute $t_{out}$ with LEC's constraint on the discharge tier. We then initialize $t_{in}$ at the empirical $(1-\kappa)$ quantile of scores.

Online update of the rule-in cutoff. In deployment, $t_{in}$ is nudged after each batch of patients (for example, each shift) by a quantile-tracking step. The cutoff rises when the flagged fraction exceeds $\kappa$ and falls otherwise. We will prove two results:

  • a long-run capacity bound for this update under arbitrary score sequences;
  • the preservation result above, stated carefully for the case where $t_{out}$ is itself periodically refreshed from delayed outcome labels in a sliding window.

Handling a capacity-limited gray zone. A second capacity limit may apply to the monitoring tier. If it binds, the only moves available are to lower $t_{in}$, which violates the urgent-capacity limit, or to raise $t_{out}$, which violates safety. The procedure then forbids the unsafe move and flags infeasibility. This makes the asymmetric cost ordering an explicit, auditable rule rather than a weight buried in a loss function.

Analysis in the rare-event regime. With baseline event rates around 1.4% (the Stanford chest-pain cohort) and target $\alpha$ an order of magnitude lower, the smallest certifiable $\alpha$ scales roughly as the inverse of the number of calibration patients falling below $t_{out}$. That scaling comes from the binomial tail. We will chart how much calibration data a given rule-out yield requires. This practical question is currently answered by eyeballing zero-event cutoffs.

Baselines

Naive zero-event cutoff. Following the practice in the ECG chest-pain study, we choose the largest cutoff with zero events on the development data. For the rule-in tier we pair it with a static retrospective quantile. It runs on the same score and splits as everything else. The file reports the resulting zero-death rule-out and an AUC gain from 0.75 to 0.80 over clinical plus troponin. It reports no out-of-sample miss rate for that cutoff, and measuring one is our first check.

Empirical cost-minimizing thresholding. This is Thresholding for Making Classifiers Cost-sensitive, which searches cutoffs by internal cross-validation and picks the centre of the widest minimum-cost valley. It beat the theoretical cost formula on 9 of 12 UCI datasets. We run it with the same folds, sweeping cost ratios because no true ratio exists. It is the strongest heuristic that needs no calibration theory.

Fixed-sequence exact binomial testing. Candidate $t_{out}$ values are scanned from strict to lenient on the calibration split. We accept a candidate while a one-sided Clopper–Pearson upper bound on the discharge-tier event rate stays below $\alpha$. This is the most direct simple way to get a valid conditional guarantee, and it is a serious competitor.

LEC-Routing, the strongest existing method. We apply LEC with both thresholds calibrated jointly and statically on labels. On TriviaQA it retained 9% more valid samples than the best UCB baseline at equal risk. At α = 0.05 its two-model routing accepted 13.6% more samples than a single model. There is no reported result on Lacuna for LEC on clinical triage data.

Online deferral with regret guarantees. HI-LCB-2D from Learning to Offload proves O(log T) regret against O(T^{2/3}) for H2T2. It is fed the same stream with a cost matrix swept over the same range.

Ablations. Two ablations isolate the key ingredient. The first replaces the conditional constraint with marginal conformal risk control (Conformal Risk Control for Ordinal Classification). The second freezes $t_{in}$ at its retrospective value.

Tuning and what counts as winning. Every method gets the same score model, calibration budget, and number of tuned hyperparameters. Beating the baselines means meeting two conditions on realized test data. The discharge-tier event rate must stay at or below α in at least the nominal fraction of random splits, which the naive and cost-search methods are expected to fail. Capacity violations under shift must fall, which static LEC is expected to fail. Both conditions must hold while the discharged fraction is within a few points of the best valid competitor.

Why not just…?

Why not just calibrate probabilities with isotonic regression and apply the decision-theoretic cutoff? This needs a cost ratio no one can supply. Calibration error metrics also carry no finite-sample guarantee on the discharge tier, as LEC argues. Learning to Offload further shows that the score-to-accuracy map collapses under domain shift (below 50% local accuracy on BreaCh). We include isotonic-plus-cutoff in the cost-search baseline sweep, so its realized discharge-tier miss rate is measured directly.

Why not just train with an asymmetric loss? Every Error has Its Magnitude improves severity-aware metrics (AsMC 56.20 vs 44.16) by changing the loss. That improvement is orthogonal to our question. It still yields a score that must be cut, and it gives no bound on test-time miss rates. Retraining whenever capacity changes is also impractical. We will run our calibration on top of a cost-weighted model, which tests complementarity rather than competition.

Why not just use exact binomial bounds with fixed-sequence testing? This is a real threat for the rule-out half. For a single constraint on rare events it may certify nearly the same $t_{out}$ as LEC. If it does, we will report that and use it, because the contribution does not rest on which valid bound certifies the rule-out cutoff. The contribution is the decoupled label-free capacity update, the preservation result, and the infeasibility rule. The comparison on discharge yield at equal α settles which bound to recommend.

Why not just use LEC's joint two-threshold calibration as is? Joint calibration fits $t_{in}$ to labels from the retrospective period. It therefore inherits exactly the 14%-to-29% flag-rate drift seen in the mammography deployment, and it cannot correct that drift until outcome labels arrive. The frozen-$t_{in}$ ablation under temporal shift measures this cost directly.

Proposed Experiments

The primary testbeds are MIMIC-IV ED and ICU mortality and routing tasks, following the cohort construction of Revisiting Clinical Outcome Prediction for MIMIC-IV and An Extensive Data Processing Pipeline for MIMIC-IV. We also use the MIMIC-II septic-shock and asystole tasks from ACDC. If access permits, we use the Stanford ED chest-pain cohort, which has the 1.4% baseline mortality that makes the rule-out question sharp. The UK Biobank CVD cohort from Risk-Stratify serves as a large, low-event primary-prevention check.

Shift is created by temporal splits, with calibration on early years and deployment on later years. We also use the BreaCh cross-domain construction from Learning to Offload as a severe stress test.

We report five metrics:

  • the realized event rate among discharged patients, and the frequency of α-violations over 500 random calibration splits;
  • the discharged fraction;
  • the gray-zone fraction;
  • per-shift capacity violations in the action tier;
  • the number of adverse events landing in each tier.

Three checks are decisive:

  • Zero-event audit. Does the naive zero-event cutoff exceed its implied miss rate out of sample, while our $t_{out}$ holds α at nominal frequency?
  • Capacity under shift. Under temporal shift, does the online $t_{in}$ keep action-tier load within κ while the frozen and LEC-joint variants overshoot, with the discharge-tier miss rate unchanged? An unchanged rate is the empirical signature of the preservation result.
  • Infeasibility rule. When a gray-zone capacity is imposed, does the procedure correctly declare infeasibility instead of drifting into unsafe discharge?

A negative result on the first check, meaning that naive zero-event cutoffs turn out to be safe in practice, would itself be worth publishing. The file currently offers no out-of-sample evidence either way.