Calibrating Rare Clinical Codes Against Their Negatives

When a model assigns diagnosis codes to a hospital discharge summary, the codes that matter most are often the ones it has barely seen. In MIMIC-IV, half of all ICD-10 codes appear in three or fewer of about 122,000 summaries. Conformal prediction is a wrapper that turns a model's scores into a set of predicted codes with a guaranteed hit rate. Its usual per-class form, "include code $c$ whenever its score clears a threshold", needs enough positive examples of $c$ to set that threshold, and for these codes there are none to spare. Existing fixes pool rare codes with related ones and share a single threshold. That only works if the pooled codes' scores live on the same scale, and raw scores from different codes, and from different models, do not.

We start from one asymmetry: a rare code has almost no positive documents but thousands of negative ones. We propose to rescale each code's score, for each model in an ensemble, into its empirical false-positive rate: the fraction of calibration documents without that code that score at least as high. On this scale, scores from rare and common codes, and from a pre-trained language model and an ontology-based synonym matcher, become directly comparable. Rare codes can then be pooled safely and ensemble members combined without learned weights. The claim is that this rescaling is the missing step that lets pooled conformal calibration work for multi-label clinical coding. It should give per-code recall guarantees on the long tail with smaller prediction sets than taxonomy pooling on a single model.

Core Claim

In multi-label coding, a prediction set with class-conditional coverage at level $1-\alpha$ means something concrete: for each code, the recall on documents that truly carry it is at least $1-\alpha$. Calibrating a per-code threshold is therefore the same as choosing a recall operating point. The difficulty is purely statistical: with two calibration positives, the empirical quantile is useless.

For model $m$ and code $c$, let $s_m(x,c)$ be the raw score. Let $N_c$ be the calibration documents lacking $c$, which number in the tens of thousands even for rare codes. Define

$$u_m(x,c) = \frac{1 + |\{x' \in N_c : s_m(x',c) \ge s_m(x,c)\}|}{1 + |N_c|},$$

the code-specific false-positive rate at which document $x$ would be flagged. The ensemble score is $u(x,c) = \min_m u_m(x,c)$, or a combination chosen on a held-out selection split. A code enters the set when $u(x,c) \le \tau_g$. The threshold $\tau_g$ is the conformal quantile of the true-code scores pooled over the positives of all codes in group $g$.

The guarantee this provides is group-level recall, and per-code recall holds only to the extent that pooled codes are exchangeable on the $u$ scale. We will not claim more. The hypothesis is that $u$ makes pooled codes far closer to exchangeable than raw scores. That would turn the group guarantee into near-per-code behaviour empirically, and it would shrink sets because no single code's poorly scaled scores force a loose shared threshold.

Relation to Prior Work

Three papers supply the conformal machinery and also define the gap. Robust Conformal Prediction for Infrequent Classes pools calibration data for rare diagnoses using the ICD or CCSR body-system hierarchy. It raises infrequent-class coverage 1.7x to 2.9x over class-conditional CP on MIMIC, but it inflates set size, uses one base model, and is formulated for single-label prediction. Class-Conditional Conformal Prediction With Many Classes clusters classes by the quantiles of their scores. It sends classes with fewer than ten examples to a fallback cluster, which is exactly the regime that dominates clinical coding. Enhancing Conformal Prediction via Class Similarity shrinks sets by penalizing classes far from the top prediction in latent space, cutting CIFAR-100 set size from 5.79 to 3.79. It was tested only on balanced vision data with single-label top-1 anchors, which do not exist in multi-label coding.

We reuse the pooling idea from the first two papers but change what is pooled. They pool raw scores; we pool negative-rescaled scores, which can be estimated for any code with negatives. Following the third, we also test grouping by similarity in a domain-pretrained embedding of code descriptions. Unlike score-quantile clustering, this works for codes with zero calibration positives.

Two papers motivate the ensemble. Mimic-IV-ICD shows that a pre-trained language model coder (PLM-ICD) wins on frequent codes while the synonym-matching network MSMN wins on rare ones, but it never combines them and reports only point predictions. Medication Extraction and Entity Linking using Stacked and Voted Ensembles on LLMs shows that mixing general and clinically pre-trained encoders beats every single member (Macro F1 0.8232 with max-logit voting). It also observes that stacking on raw confidences overfits to overconfident members, which forced the authors to pass one-hot votes instead. Our false-positive-rate rescaling is a direct answer to that calibration problem: members are compared on a shared empirical scale rather than on their own confidences.

Multi-model Ensemble Conformal Prediction in Dynamic Environments selects among ensemble members to minimize set size, but it does so over time under covariate shift and with marginal coverage only. We select over code groups, with class-conditional targets.

THCM-CAL applies standard split conformal prediction to MIMIC diagnosis prediction. It is our clearest example of what marginal coverage hides, because a predictor can reach 90% overall coverage while dropping rare codes. Finally, Automated Medical Coding on MIMIC-III and MIMIC-IV constrains our evaluation: it shows that per-code threshold tuning and stratified splits matter a great deal, and that Macro F1 has been miscomputed in earlier work.

Research Program

The ensemble has three members, chosen for complementary pre-training rather than for size. The first is PLM-ICD with its standard encoder. The second is the same architecture with a clinically pre-trained encoder, GatorTron-base as described in GatorTron and GatorTronGPT, or BioClinicalBERT if compute binds. The third is MSMN, the synonym-matching network.

All three are trained on a stratified training split. We hold out three disjoint partitions: one for selecting the combination rule, one for conformal calibration, and one for testing. Codes with no training positives are scored only by MSMN through its synonym matching, and they are reported separately.

We will build the rescaling, the group-pooled thresholds, and three grouping schemes. The first two are the ICD-10 chapter or block and the CCSR body system, both as in DGCP. The third clusters code-description embeddings taken from the clinical encoder, with the number of groups chosen on the selection split so that each group has at least $m$ calibration positives.

We will also analyze, per frequency bucket, how heterogeneous the pooled positive scores are before and after rescaling. We will measure this with the spread of per-code empirical recall at the shared threshold. This is the mechanism check: if rescaling does not reduce the spread, the method's advantage, if any, comes from somewhere else.

Baselines

The naive reference is marginal split conformal prediction on the single best member, the approach used in THCM-CAL. It should hit nominal coverage overall while undercovering the tail, and every method must beat it on tail recall.

The second baseline is per-code threshold tuning in the style of Automated Medical Coding on MIMIC-III and MIMIC-IV. Thresholds are tuned to a recall target of $1-\alpha$ on the same calibration data, with no guarantee. This tests whether conformal machinery adds anything beyond careful thresholding.

The third is classwise conformal prediction, which has the correct target but is noisy, with effectively infinite thresholds for codes that have one or two positives. The fourth is Clustered CP from Class-Conditional Conformal Prediction With Many Classes, including its fallback cluster.

The strongest competitor is DGCP with body-system grouping from Robust Conformal Prediction for Infrequent Classes, adapted to per-code multi-label thresholds. Its own paper reports this variant as having the best macro coverage on MIMIC. The file records only the relative 1.7x to 2.9x coverage gain over class-conditional CP; no reported absolute coverage or set-size result was found on Lacuna for DGCP. We will therefore rerun it ourselves. It will run on the best single member and also on a raw-probability-averaged ensemble, so that the ensemble alone is not credited to our method.

All methods use identical trained models, splits, $\alpha \in \{0.1, 0.2\}$, and minimum group size $m$. Each gets the same budget for its tuning knobs on the selection split. Beating DGCP means one of two outcomes, both read in the rare buckets (codes with at most ten training positives): matching its per-code recall coverage with a smaller average number of codes per document, or improving the worst-bucket coverage gap at equal set size.

Three ablations isolate the ingredients. The first removes the rescaling, pooling raw scores within the same groups. The second keeps the rescaling but uses a single model. The third swaps the domain-pretrained description embedding for a general-encoder embedding, testing whether domain pre-training matters for grouping specifically.

Why not just…?

Why not just tune per-code thresholds? For frequent codes this may be nearly as good, and we expect it to tie there. For codes with two calibration positives, a tuned threshold is fit to two numbers and gives no statement about future recall. The thresholding baseline will show whether its realized test recall on rare codes falls below target, and by how much it varies across random calibration splits. If it does not fall short, the honest conclusion is that guarantees add little in this regime, and we will report that.

Why not just run DGCP on the single best model? This is the real threat. The ensemble might add nothing once pooling is in place. Mimic-IV-ICD's frequency-dependent split between PLM-ICD and MSMN suggests it will, but only if the members' scores can be pooled together, and that is what rescaling provides. The single-model rescaling ablation and single-model DGCP settle this directly.

Why not just average the ensemble's probabilities and then pool? Averaging is the obvious ensemble, but raw probabilities from a language model and a synonym matcher are not on comparable scales. The stacking results in the medication-extraction paper show this kind of confidence mismatch hurting fusion. The averaged-ensemble DGCP baseline measures how much this costs. If averaging matches our combination, the contribution reduces to the rescaling alone, which is still testable through the raw-score ablation.

Why not just use one larger clinical model such as GatorTron? A larger encoder may improve head-code scores. However, the long-tail weakness reported for pre-trained coders is structural, and the extra capacity is not shown to fix it. Including the clinically pre-trained encoder as one member lets us read its standalone conformal performance by frequency bucket.

Proposed Experiments

The main benchmark is MIMIC-IV-ICD10 from Mimic-IV-ICD, where point models reach Macro AUC up to 97.07% but Macro F1 of only 5.42%. That gap is a sign that per-code scores rank well within a code but are badly scaled across codes, which is the situation the rescaling targets. We replicate on MIMIC-IV-ICD9 and on the 1,054-class MIMIC-IV ICD split used by DGCP. Point-prediction quality is reported with the corrected per-class Macro F1.

Conformal outputs are reported as per-code recall coverage and coverage gap, broken down by training-frequency bucket. We also report average set size and false positives per document, all with confidence intervals over twenty random calibration and test resplits.

The decisive check comes in two parts. First, in the rare buckets, the full method must beat body-system DGCP on set size at matched coverage. Second, the raw-score ablation must lose most of that gain, while the measured spread of per-code recall within groups must shrink after rescaling. If set sizes shrink but the spread does not, the mechanism we propose is wrong and the gain needs another explanation. A secondary check is whether description-embedding groups beat ontology groups. Enhancing Conformal Prediction via Class Similarity found latent similarity beating human taxonomy, while DGCP found body systems best. Our clinical setting decides between them.