A model that assigns diagnosis codes to hospital discharge notes should be allowed to say "I am not sure about this code" and hand that decision to a human coder. The standard way to do this is selective prediction: the model predicts only when its confidence exceeds a threshold and defers otherwise. A single threshold shared by all codes is known to fail, because codes differ in how spread out and confusable their examples are. The obvious fix is a separate threshold per code, tuned on held-out data. That fix breaks down where it matters most. In MIMIC-IV with ICD-10 codes, half of the 26,096 codes appear in three or fewer summaries (Mimic-IV-ICD), which leaves no data to tune a threshold for those codes.
We propose to predict the abstention threshold for a rare code from where that code sits in the representation space of a language model pretrained on medical text. The recipe has three steps. We measure, on frequent codes, how each code's ideal threshold relates to cheap geometric descriptors: how crowded the code's neighborhood of related codes is, how widely its few examples spread, and where it sits in the ICD hierarchy. We then transfer that relationship to rare codes. Finally, we blend the transferred estimate with whatever sparse evidence the rare code has. The scientific bet is that domain-specific pretraining makes this geometry a more faithful map of each code's error behavior than general-domain pretraining does. We test the bet directly by holding the architecture fixed and changing only the pretraining corpus.
Core Claim
Class-specific thresholds are known to help. The open question is how to set them without per-class calibration data. Our claim is that a per-class threshold is largely predictable from label geometry, and that this predictability is a measurable, previously unstudied benefit of domain pretraining. Two concrete predictions follow. First, a regression from geometric descriptors to per-code thresholds, fit on frequent codes, should explain much more variance on held-out frequent codes when the descriptors come from a medically pretrained encoder than from a general one. Second, using that regression as a prior for rare codes should cut the number of rare codes whose realized error exceeds the target budget, at equal or better coverage, compared with every simpler way of pooling rare codes.
The directions play different roles. Selective prediction at the level of individual labels is the task. Class-specific thresholding is the mechanism. Domain-specific pretraining is the experimental variable whose effect on threshold transferability we measure.
Relation to Prior Work
Three papers motivate the proposal by showing that one global threshold fails. Revisit Overconfidence for OOD Detection sets class-dependent thresholds for intent detection and gains 10.44% out-of-distribution F1 over maximum softmax probability on CLINC-Full. It uses a general fine-tuned encoder and frames the problem as detecting unfamiliar inputs, not trading coverage against risk. Improving Cascade Routing shows that thresholds grouped by attribute type route 19.3% instead of 26.4% of traffic to a large model at matched precision. Its groups are coarse data types with plenty of data each. Reducing Class-Wise Performance Disparity via Margin Regularization ties per-class error to feature spread through a class-sensitive generalization bound. That is the theoretical reason we expect spread to predict thresholds, but the paper works on closed-world vision models and has no abstention mechanism.
Two papers are the closest competitors on the task itself. Uncertainty-aware abstention in medical diagnosis based on medical texts introduces rejection of individual labels on MIMIC-IV top-50 ICD-10 codes with the HUQ-2 uncertainty score, but applies one shared threshold across labels. It also reports that Monte Carlo dropout can do worse than random guessing there, which constrains our design away from sampling-based uncertainty. Act or Defer, known as STRATCP, enforces per-class error budgets with stratified conformal prediction on a frozen EHR model. It shows that marginal conformal prediction at α=0.10 selected mostly wrong actions for pancreatic cancer while STRATCP kept coverage above 99%. STRATCP needs calibration samples for every class and is agnostic to representation. We keep its per-class error-budget goal and replace its per-class data requirement with a geometric prior.
Domain pretraining supplies the method variable. BioBART continually pretrains BART on PubMed and itself flags a gap when transferring from literature to clinical notes. That gives us a controlled trio of encoders: general BART, BioBART, and BioBART further pretrained on clinical notes. Selective Omniprediction and Fair Abstention proves that for multicalibrated predictors the optimal abstention region is a single interval of predicted probability. Our method is consistent with that result applied one code at a time, since each code still gets a single cutoff. The difference is that the cutoff location varies across codes, and we estimate it where calibration data is absent.
Research Program
For each code $c$, let $\tau_c$ be the smallest confidence at which accepted positive predictions for $c$ meet a target error budget α. On frequent codes $\tau_c$ can be estimated empirically. We compute a descriptor vector $z_c$ for every code, including codes with no examples. The descriptors are neighborhood crowding, measured as the density of other code-description embeddings around code $c$'s description embedding; the spread of $c$'s training examples when they exist; log frequency; and depth and sibling count in the ICD tree.
We fit a regressor $h$ from $z_c$ to $\tau_c$ on frequent codes. We then set each code's threshold by shrinkage:
$$\hat\tau_c = \frac{n_c}{n_c+\kappa}\,\tau_c^{\mathrm{emp}} + \frac{\kappa}{n_c+\kappa}\,h(z_c),$$
Here $n_c$ is the number of calibration positives for code $c$, $\tau_c^{\mathrm{emp}}$ is its empirical threshold, and $\kappa$ is a single pooling strength chosen by cross-validation. The weighting is the standard empirical-Bayes form in which evidence overrides the prior as samples accumulate.
The classifier head, confidence score, and training are identical across encoders. Only the backbone's pretraining corpus changes.
The analysis has two parts. The first measures how well $h$ predicts held-out frequent-code thresholds for each encoder. The second measures how often rare codes violate their error budget. Rare codes cannot each be audited, so we report violation on pooled tail buckets and on cross-fitted splits. We make no per-code guarantee for codes with three examples, and we say so plainly.
Baselines
The naive reference point is a single global threshold on the same confidence score, tuned to meet α on average. This is the practice in Uncertainty-aware abstention and in EHRSQL. Beating it means fewer codes over budget at matched overall coverage.
The strongest existing method for label-wise clinical abstention is HUQ-2 with its shared threshold. We will run it on the same encoder, splits, and tuning budget. We found no reported result on Lacuna for HUQ-2 on MIMIC-IV top-50 in the form of FR-AUC numbers, so we will reproduce it. As a benchmark-level anchor, the best models on full MIMIC-IV-ICD10 reach Macro AUC 97.07% but Macro F1 only 5.42% (Mimic-IV-ICD). That gap is exactly why rare-code decisions need a defer option.
The strongest per-class competitor is STRATCP-style stratified conformal selection (Act or Defer) applied to label-wise decisions. For codes with too few positives it falls back to the global threshold. It receives the same calibration split. Beating it means matching it on frequent codes, where it should be near-optimal, and reducing violations on the tail.
Two pooling baselines isolate our ingredient. The first gives one threshold per frequency bucket. The second pools each rare code with its ICD parent. The key ablation replaces the geometric descriptors from the medical encoder with those from general BART while keeping everything else fixed. If that swap does not hurt, the domain-pretraining claim fails.
Why not just…?
Why not just pool rare codes by frequency bucket? This is a real threat. If a code's error behavior is driven mainly by how few examples it has, frequency alone captures most of the signal. The evidence against it is indirect. Margin Regularization finds per-class accuracy ranging from 100% to 16% on balanced ImageNet, so spread matters independently of frequency. Our regression reports the variance explained by frequency alone versus frequency plus geometry. If geometry adds nothing, we will report the bucket method as the answer.
Why not just pool through the ICD hierarchy? The ontology is free and clinically meaningful. Sibling codes, however, can differ sharply in how confusable they are: a "specified" code and its "unspecified" sibling are an example. Hierarchy depth and sibling count are inputs to $h$, and parent pooling is a baseline. The comparison settles whether learned geometry adds anything beyond the tree.
Why not just use a better domain model and one threshold? Domain pretraining might shrink differences between codes enough that one threshold suffices, which is itself Gap 1 in the file. We test it directly by measuring the spread of per-code $\tau_c$ under each encoder. If the medically pretrained encoder collapses that spread, the proposal's value shifts to documenting that finding, and we will present it that way.
Why not just prompt a large general model to judge its own answers? Leveraging Data to Say No found that a 72B model used as a judge was worse at selective prediction than a 16M-parameter model with retrieval memory, and asking such a model about 26,000 codes per note is impractical. We therefore treat this only as a small spot-check on top-50 codes.
Proposed Experiments
The main benchmark is MIMIC-IV-ICD10 in two versions: the top-50 codes, for comparability with HUQ-2, and the full label set, for the tail. Encoders are BART, BioBART, and BioBART with continued pretraining on MIMIC-III notes, which that paper names as future work.
We report four metrics. The first is the fraction of codes whose accepted-prediction error exceeds α, broken down by frequency bucket. The second is label-level coverage. The third is FR-AUC. The fourth is the held-out R² of $h$ for each encoder.
A shift test uses the veterinary notes from DeepTag. It reports weighted F1 of 0.824 on the in-site CSU notes and 0.634 on the out-of-site private-practice notes. We fit thresholds on CSU and ask whether geometry-transferred thresholds degrade less on the private-practice set than empirically tuned ones.
Three checks are decisive. The ordering BART < BioBART < clinical BioBART in threshold predictability would support the domain-pretraining claim. A drop in tail violations beyond both pooling baselines would support the method. A failure on either check would falsify the corresponding part of the claim.
We also note a constraint for any deployment. On the Limits of Selective AI Prediction showed that when the model stays silent, clinicians read the silence as a negative and miss 18% more diagnoses. Deferred codes must therefore be shown to the human explicitly as "undecided." We state this requirement but do not run a user study.
References
- Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification
- Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold
- Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence
- Reducing Class-Wise Performance Disparity via Margin Regularization
- Uncertainty-aware abstention in medical diagnosis based on medical texts
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- Selective Omniprediction and Fair Abstention
- EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
- Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction
- DeepTag: inferring diagnoses from veterinary clinical notes
- On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making