Many outcomes we want to predict from domain text are mostly zero. Examples are ICU hours for emergency-department patients, where most patients never reach the ICU, and the error rate of a speech transcript, where many transcripts are perfect. A hurdle model handles this in two stages. The first stage is a classifier that asks "is the outcome above zero?" The second stage is a regression that asks "given that it is positive, how large is it?" Two other tools are standard in this setting: domain-pretrained encoders such as PubMedBERT, and ensembles of several such models. Combining them seems easy: put a hurdle head on each of several clinical encoders and average the results. That averaging step is where the difficulty lies. Averaging final expected values mixes the probability of being nonzero with the conditional size. Averaging full predictive distributions is valid, but it forces every model to have the same influence in both stages. Our hypothesis is that the two stages need different ensembles. Deciding whether an event happens and estimating how large it is draw on different information, so a model that is reliable at one can be unreliable at the other.
We propose to fit separate combination weights, and possibly separate member sets, for the hurdle stage and the magnitude stage of an ensemble of domain-pretrained text encoders. We also propose to measure directly whether the members' errors are correlated differently in the two stages. If they are, stage-wise stacking should beat both averaging expected values and single-mixture ensembling. If they are not, a simple seed ensemble should be enough. Either result is useful, and the experiment tells the two apart.
Core Claim
A single member $k$ of the ensemble predicts a probability $\pi_k(x)$ that the outcome $y$ is positive for input text $x$. It also predicts a density $f_k(y \mid x, y>0)$ over the positive values. The usual way to combine $K$ members with weights $w_k$ is to mix their full distributions. That produces a pooled positive probability $\sum_k w_k \pi_k$ and a positive-part density in which member $k$ gets weight $w_k \pi_k / \sum_j w_j \pi_j$. The same weights therefore control both stages. A model that is confident an event will happen automatically gains influence over the event's predicted size, whether or not it is good at estimating size. We propose instead
$$ \bar\pi(x) = \sum_k a_k \pi_k(x), \qquad \bar f(y \mid x, y>0) = \sum_k b_k f_k(y \mid x, y>0), $$
where $a$ and $b$ are separate simplex weights fit on held-out data. We fit $a$ by log loss on the zero/nonzero indicator and $b$ by log score on the positive examples only. The combined model is still a valid hurdle distribution, and its two stages are fit with separate losses. A small meta-network can also make $a$ and $b$ depend on the input $x$. We claim that $a$ and $b$ will differ systematically, and that this difference drives the gain.
The empirical prediction is specific. Broadly pretrained or clinically pretrained encoders should earn weight in the hurdle stage, because deciding whether an event occurs depends on common triage language. Encoders with specialized vocabularies should earn weight in the magnitude stage, because estimating severity depends on precise technical terms. This reuses the division of labor that A Deep Fusion Model for Domain Adaptation in Phrase-based MT found when it fused separately trained in-domain and out-of-domain translation models. We move that idea from translation fusion to the two stages of a hurdle model.
Relation to Prior Work
Prior work covers each pair of these directions but not the combination. Neural Zero-Inflated Quality Estimation Model For Automatic Speech Recognition System is the closest work. It places a zero-inflated Beta head on one pretrained Speech-BERT encoder to predict word error rate. It reports an MAE of 9.87 on CHiME-3, compared with 22.2 for a single-stage baseline. That result motivates the hurdle head, but the paper never combines several encoders. Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation and DNet: Distributional Network for Distributional Individualized Treatment Effects show that neural two-stage heads beat single-stage losses on structured features. We reuse their head design. Both train a single network, so neither addresses combination. Towards more realistic climate model outputs: A multivariate bias correction based on zero-inflated vine copulas does handle zero mass inside an ensemble. However, it post-processes physical simulations offline and does not combine learned predictors.
On the encoder side, Revisiting Clinical Outcome Prediction for MIMIC-IV supplies the member models and a key observation. In that study, PubMedBERT reaches 86.86% AUROC but only 17.24% PR-AUC on emergency-department diagnoses. The best single domain encoder therefore still fails on rare events, which is where a well-built hurdle stage should help. Leveraging Deep Representations of Radiology Reports in Survival Analysis for Predicting Heart Failure Patient Mortality shows that BlueBERT features carry signal for continuous clinical risk. This supports using these encoders for the magnitude stage.
Evidence on ensembles both motivates and constrains the design. Technical Report on Shared Task in DialDoc21 combined RoBERTa and ELECTRA, which have different pretraining objectives, and found they contributed complementary signal. A Comprehensive Survey on Imbalanced Data Learning argues for separating representation learning from the prediction head, and for cost-sensitive ensembles, in rare-event settings. The survey also implies a warning: members that share failure modes give little benefit when combined. That warning is why error-correlation analysis is a central measurement here rather than an afterthought.
Research Program
The first component is the set of members. We fine-tune hurdle heads on six encoders from the MIMIC-IV comparison: PubMedBERT, BioBERT, ClinicalBERT, BlueBERT, DischargeBERT, and a general RoBERTa, each with three random seeds. Every head has a Bernoulli output for the hurdle and a positive-support distribution for the magnitude. We use a lognormal distribution, or a zero-truncated negative binomial for count targets. As a sensitivity check, we also use a non-crossing quantile head in the style of DNet, to cover heavy tails.
The second component is the combination methods. We compare four ways of combining the same predictions. The first averages point predictions $\pi_k \mu_k$, where $\mu_k$ is member $k$'s predicted positive mean. The second is the standard single-weight mixture. The third is our stage-wise stacking with fixed per-stage weights $a$ and $b$. The fourth is input-dependent stage-wise stacking, where a small meta-network sees the members' predictions plus note-level metadata such as note length and department.
The third component is the diagnostics. On held-out data we compute the correlation of member errors separately for each stage. For the hurdle stage the error is the residual of the logit. For the magnitude stage it is the log-score residual on positive cases. From these correlations we derive an effective number of independent members per stage. The central analysis tests whether this effective number differs across stages and whether the difference predicts where stacking helps.
Baselines
The first reference point asks whether the hurdle structure matters at all. It is a single PubMedBERT with a single-stage Tweedie or MSE regression head. PubMedBERT is the strongest single encoder in Revisiting Clinical Outcome Prediction for MIMIC-IV.
The strongest single-model competitor is PubMedBERT with a hurdle head. This is the design of Neural Zero-Inflated Quality Estimation Model For Automatic Speech Recognition System moved to clinical text. It is the natural thing a practitioner would build today.
The strongest ensemble competitor is the standard full-distribution mixture of all eighteen hurdle members, with one set of weights fit on held-out data. It uses exactly the same members and the same held-out split as our method, so any difference comes from separating the stage weights.
We also include deep fusion as in A Deep Fusion Model for Domain Adaptation in Phrase-based MT. In this variant, the final hidden layers of PubMedBERT and RoBERTa are concatenated and fine-tuned with one shared hurdle head. It tests whether combining at the representation level beats combining at the output level.
Two ablations isolate our key ingredient. The first ablation forces $a = b$ while keeping everything else fixed. The second replaces the heterogeneous members with an eighteen-seed PubMedBERT ensemble, which removes pretraining diversity while keeping the member count.
All methods share the same splits, the same hyperparameter search budget per member (the same number of trials), and the same held-out set for fitting weights. Ensembles are compared at matched numbers of forward passes. We also report a single model distilled from the best ensemble, to address the concern about inference cost raised in APEX: A Network-Native Time-Series Foundation Model for Forecasting and Anomaly Detection for Wireless Edge Operations.
For our constructed MIMIC-IV targets, no reported result was found on Lacuna. The closest published figure applies only to the hurdle-like classification task: PubMedBERT reaches 17.24% PR-AUC on emergency-department diagnoses. For CHiME-3, the number to beat is the MAE of 9.87 reported for Speech-BERT with a zero-inflated Beta head.
We count our method as winning if it improves the combined continuous ranked probability score (CRPS, a proper scoring rule for full predictive distributions) over the single-weight mixture. The improvement must exceed the spread across three repeated splits. It must also come without worse hurdle calibration.
Why not just…?
Why not just average the members' expected values? This is the most common practice, and it is cheap. Averaging $\pi_k \mu_k$ can give a reasonable mean. However, it produces no predictive distribution, which rules it out when calibrated intervals or zero probabilities are needed, as in ICU capacity planning. It also cannot say which stage the gain comes from. We still include it, and we report MAE as well as CRPS so that it has a fair chance where means are all that matter.
Why not just use one strong domain encoder with a hurdle head? The evidence from MIMIC-IV cuts both ways. PubMedBERT's gap between AUROC and PR-AUC shows blind spots on rare events. It does not show that other encoders fill those blind spots. If the cross-member error correlations in the hurdle stage are close to one, this alternative wins, and we will report that. The error-correlation diagnostic settles the question before any claim about stacking is made.
Why not just ensemble seeds of PubMedBERT? This is the most serious threat. Seed ensembles often capture most of the benefit of ensembling. The eighteen-seed ablation tests this directly. If it matches the heterogeneous stage-wise ensemble, then pretraining diversity is not the useful ingredient, and the contribution shrinks to stage-wise weighting alone. That narrower result would still be worth reporting.
Why not just prompt or lightly fine-tune a large general model? Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records reports 0.50 AU-ROC for zero-shot Llama 2 13B and 0.61 for Meditron 7B on mortality prediction. Fine-tuned domain models reach 0.87–0.88. This makes a zero-shot model a weak competitor for predicting numeric magnitudes. We therefore include it only as a single Meditron member with a fine-tuned hurdle head, so that its stage weights can show whether it contributes anything.
Proposed Experiments
The primary benchmark uses the MIMIC-IV emergency-department and ICU splits from COP-IV. From these we derive three zero-heavy targets, each predicted from the notes available at emergency-department disposition. The first is ICU hours within 72 hours, which is zero for patients never admitted to the ICU. The second is the number of procedures during the stay. The third is days on vasopressors. Metrics are CRPS and mixed discrete–continuous log score for the full distribution, PR-AUC and expected calibration error for the hurdle stage, MAE on positive cases for the magnitude stage, and overall MAE.
The secondary benchmark is CHiME-3 word-error-rate estimation. Here the members are Speech-BERT-style encoders pretrained on different speech data. This benchmark lets us compare directly against the reported MAE of 9.87.
The decisive checks fall into three parts. First, the fitted weights $a$ and $b$ should differ in a way that is stable across splits, measured by the distance between them relative to bootstrap spread. The per-stage effective number of members should also differ. Second, if both hold, stage-wise stacking should beat the $a = b$ ablation on CRPS, with most of the gain in the positive tail. Third, the heterogeneous ensemble should beat the matched seed ensemble.
If the first check fails, the stages do not need different ensembles. We would then conclude that the standard mixture is adequate for hurdle models, which is itself a clean negative result that practitioners can use.
References
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- Neural Zero-Inflated Quality Estimation Model For Automatic Speech Recognition System
- Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation
- DNet: Distributional Network for Distributional Individualized Treatment Effects
- Towards more realistic climate model outputs: a multivariate bias correction based on zero-inflated vine copulas
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- Leveraging Deep Representations of Radiology Reports in Survival Analysis for Predicting Heart Failure Patient Mortality
- Technical Report on Shared Task in DialDoc21
- A Comprehensive Survey on Imbalanced Data Learning
- APEX: A Network-Native Time-Series Foundation Model for Forecasting and Anomaly Detection for Wireless Edge Operations
- Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records