Fixing Rare-Code Decisions in Clinical Coding Without Retraining

Automatic medical coding assigns standardized diagnosis codes (ICD-10) to a hospital discharge note. It is a clear case of a specialized domain whose labels follow a long tail. On the MIMIC-IV-ICD10 benchmark there are 26,096 codes, and half of them appear in three or fewer training notes. The benchmark reports a striking pair of numbers. The best models reach a macro-averaged AUC of up to 97.07%, yet their best macro-averaged F1 is only 5.42% (Mimic-IV-ICD). AUC asks whether a model ranks notes with a code above notes without it. F1 asks whether the model actually says "yes" at the right moments. A model that ranks well but rarely commits to rare codes fails in the decision step, not necessarily in what it has learned.

We propose to test that reading directly and to repair it after training. Every code gets its own decision offset, a number subtracted from its output score before the yes/no cut. The difficulty is that most codes have no validation examples from which to tune that offset. So we estimate the offset for rare codes by borrowing information from three places. The first is the model's own average output on unlabeled notes, which measures how far its learned prior drifts from the true code frequency, following the "effective prior" idea of Prior2Posterior. The second is the code's neighbors in the ICD hierarchy. The third is the code's training count. If the claim holds, much of the gap between head-code and tail-code performance in domain-specialized coders can be closed with no new data, no retraining, and almost no compute.

Core Claim

On extreme multi-label clinical coding, a large part of the macro-F1 collapse comes from badly placed per-code decision offsets rather than from poor note representations. A hierarchy-pooled, effective-prior offset estimator recovers a substantial fraction of the gap between the default 0.5 threshold and an oracle per-code threshold, including on codes that have zero validation positives.

A second claim ties this to domain specialization. The more aggressively a backbone is specialized by full-parameter domain fine-tuning, the larger the gap becomes between its effective prior and the true frequency on tail codes. Light, parameter-efficient adaptation narrows that gap. This carries the "heavy fine-tuning hurts" finding of Long-Tail Learning with Foundation Model from single-label vision into multi-label clinical text.

We are explicit about what would falsify the first claim. Macro AUC is insensitive to how rare a code is. A code with prevalence near one in ten thousand can have AUC 0.97 and still have poor precision at every threshold. If an oracle that picks each code's best threshold on the test set itself still leaves macro-F1 low, the bottleneck is representation. In that case the proposal pivots to the decoupled-head experiments below.

Research Program

We use classifier-style coders that output one sigmoid score per code. The first is PLM-ICD, a biomedical pretrained encoder with segment pooling and per-code attention (PLM-ICD). The second is MSMN, a synonym-matching model that grounds codes in their dictionary descriptions; the benchmark paper finds it beats PLM-ICD on the tail while PLM-ICD wins the head (Mimic-IV-ICD). The third is a compact biomedical LLM backbone (BioMistral 7B, BioMistral) with a per-code attention head. We train this backbone two ways: full fine-tuning, and LoRA with fewer than 1% of parameters trainable in the style of LIFT+.

The first step is diagnosis. For each trained model and each code $c$, we compute the code's training frequency $n_c$. We also compute the model's mean predicted probability $\bar p_c$ over a large pool of notes, which requires no labels. The quantity $\log \bar p_c - \log (n_c/N)$, where $N$ is the number of training notes, is the multi-label analogue of the "bias gap" in Prior2Posterior. We plot it against frequency tier, separately for each backbone and each tuning regime. We also measure classifier weight norms per code, which tests whether the head-norm imbalance found in vision (The Devil Is in Classification) and in single-label text (Making Pretrained Language Models Good Long-tailed Learners) also appears with tens of thousands of correlated labels.

The second step is the correction. Each code's score $z_c(x)$ on note $x$ becomes $z_c(x) - \hat b_c$, where

$$\hat b_c = \lambda_c\, b_c^{\mathrm{val}} + (1-\lambda_c)\, g\big(\log n_c,\ \log \bar p_c,\ \hat b_{\mathrm{parent}(c)}\big).$$

Here $b_c^{\mathrm{val}}$ is the F1-maximizing offset tuned on validation, which is defined only when $c$ has validation positives. The function $g$ is a small regression fit on codes that do have enough validation positives. It maps the code's frequency, its effective prior, and its parent category's offset to a predicted offset. The weight $\lambda_c \in [0,1]$ grows with the number of validation positives for $c$, so frequent codes use their own tuned offset and rare codes lean on the pooled prediction.

This design differs from Prior2Posterior in two ways. It works on independent sigmoids rather than one softmax. It also uses the ICD tree to share strength among rare sibling codes, which directly targets the "hierarchy-level errors" reported by Hou et al..

The third step tests whether the correction transfers from frequent codes to rare ones. We take medium-frequency codes and artificially thin their validation positives down to 0–3 examples. We then check whether $g$ predicts their true oracle offsets. This guards against the main risk: the regression extrapolating poorly from codes with many examples to codes with very few.

Baselines

The naive reference point is each trained model thresholded at 0.5 for every code, which is how the reported 5.42% macro-F1 arises. The weakest correction is a single global threshold tuned on validation. The textbook frequency correction subtracts $\log n_c$ from every logit. Prior2Posterior argues this correction is miscalibrated because the learned prior drifts from the empirical counts. Including it tests whether the drift matters here.

The strongest post-hoc competitor is per-code threshold tuning on validation, using fallback to the global threshold for codes with no validation positives. It uses exactly the same checkpoint and validation split as our method. We beat it only if we gain macro-F1 on the tail tiers, specifically codes with 1–3 and 0 training notes, without losing micro-F1.

The strongest retraining competitor is decoupled classifier retraining. Following The Devil Is in Classification, we freeze the backbone and retrain only the per-code head with class-balanced sampling. A cumulative variant follows BBN. This baseline gets the same backbone checkpoints and a head-only tuning budget matched in GPU hours to our regression fit plus its search. If it clearly beats our method, the problem lives in head weights rather than offsets. That would be a useful finding in itself.

The strongest existing tail-oriented model is MSMN as benchmarked in Mimic-IV-ICD. We also run a validation-tuned ensemble of PLM-ICD and MSMN. For reported numbers, the best full MIMIC-IV-ICD10 macro-F1 is 5.42% and macro AUC is up to 97.07%, both from Mimic-IV-ICD. No reported result was found on Lacuna for per-tier macro-F1 on MIMIC-IV-ICD10, for per-code threshold tuning, or for decoupled heads on this benchmark, so we will produce these ourselves.

Two ablations isolate the key ingredient. One removes the hierarchy term from $g$. The other removes the effective-prior term $\bar p_c$, leaving frequency only.

Why not just…?

Why not just tune a threshold per code on the validation set? Half of the ICD-10 codes have at most three training notes, so most have zero or one validation positive. Their tuned thresholds are either undefined or noise. Our method reduces to exactly this baseline for well-populated codes, so the comparison is clean: any gain must come from the tail tiers. The thinned-medium-code experiment shows directly how noisy single-example tuning is.

Why not just generate synthetic notes for rare codes? Synthetic Clinical Notes for Rare ICD Codes generated 90,000 LLM-written notes and moved MIMIC-III macro-F1 from 12.3 to 13.2. That approach costs a great deal of generation compute and never touches the decision rule. We run our correction on MIMIC-III with GKI-ICD-style or PLM-ICD checkpoints to compare against this +0.9 directly. The two approaches are also complementary, so we test the correction applied on top of a synthetic-data model.

Why not just use MSMN, or ensemble it with PLM-ICD? This is a real threat. Dictionary grounding may give tail codes better ranking that no offset can match. The experiment settles it: if PLM-ICD with our correction closes most of its tail gap to MSMN, then MSMN's reported tail advantage was largely a calibration effect. If it does not, the result points to representation, and the ensemble becomes the recommended system.

Why not just retrain the head with balanced sampling? Head retraining may fix weight-norm skew that offsets cannot fix, because it changes the direction of each code's classifier and not only its intercept. We run it as a full baseline. A tie at much lower cost still favors the post-hoc method; a clear loss redirects the program.

Proposed Experiments and Decisive Checks

The main benchmark is MIMIC-IV-ICD10 with the standard splits from Mimic-IV-ICD. MIMIC-IV-ICD9 and MIMIC-III serve as secondary benchmarks, the latter for comparability with PLM-ICD (59.8% micro-F1) and the synthetic-data work. We report macro-F1 and micro-F1, precision at k, and macro-F1 by training-frequency tier (more than 100, 11–100, 4–10, 1–3, and 0 notes). We also report the rate of sibling-code errors, meaning predictions in the correct 3-character category but with the wrong full code.

The first decisive check is the oracle ceiling: per-code thresholds tuned on test. If it stays near 5–10% macro-F1, the calibration hypothesis is largely wrong. The second check is the fraction of the gap between default thresholding and the oracle that our estimator recovers on 0–3 shot codes, compared with per-code validation tuning. The third check concerns specialization: whether full fine-tuning of BioMistral shows a larger tail bias gap and a larger post-hoc gain than LoRA tuning. That outcome would show the specialization step itself creates the tail miscalibration.

We exclude generative coders such as the fine-tuned Llama models of Hou et al.. Their 69.20% top-1 exact match is measured on a 2,211-note subset with a different metric and is not comparable. Extending offset correction to code-generating decoders is left as follow-up.