Tier-Wise Decoupling for Rare ICD Codes Under Dataset Shift

Automated medical coding assigns diagnosis and procedure codes to long hospital discharge notes. It has a long-tail problem that headline numbers hide. On the MIMIC-IV ICD-10 benchmark, the best models reach a macro AUC of 97.07% but a macro F1 of only 5.42%, and half the codes appear three or fewer times (Mimic-IV-ICD). Every strong coder is also trained and tested inside a single database, so no one knows how the rare-code predictions hold up when the notes come from a different era or institution. This note proposes to answer a simple question that the field has not separated cleanly. When a hierarchical coder fails on rare codes, and fails further under dataset shift, is the damage in the learned text representation or only in the final per-code classifiers? The answer determines whether long-tail coding needs a new training scheme or just a cheaper fix applied at the end.

The method we test is decoupled training, meaning the representation is learned in one stage and the classifiers are fitted in a second stage on a frozen or partly frozen encoder. We apply it to the hierarchical encoders now standard in coding. These split a note into chunks, encode each chunk with a pretrained biomedical language model, and pool the chunk outputs into a document representation. A hierarchical encoder gives decoupling a choice that flat vision backbones do not: which tier to freeze. Our claim is that the representation should be shaped in stage 1 by supervision that is not sparse, namely three-character parent categories plus frequent codes. The chunk tier should then be frozen. Rare leaf-code classifiers should be fitted as small corrections to their parent's classifier. We predict that this matters most out of distribution. Within a dataset, simply re-fitting classifiers on an end-to-end backbone should recover much of the rare-code gap. Across datasets, it should not, because frequent-code gradients have pushed site- and era-specific cues into the backbone. If that prediction fails, the result is still useful: it says the cheap post-hoc fix is enough.

Core Claim

The idea is new at three points. It applies decoupled training to a multi-tier document encoder rather than a flat backbone. It uses the label ontology, rather than self-supervision or resampling, as the stage-1 signal. It also makes cross-dataset rare-code performance the primary outcome rather than an afterthought.

The closest work splits into two groups. The first supplies the method. Two-phase training from brain-lesion segmentation balances the data for feature learning, then freezes the feature extractor and re-fits the last layer on the true prevalence (Deep Learning Trends for Focal Brain Pathology Segmentation in MRI). That work was tested only on convolutional image models with no shift. Compositional Feature Alignment shows that staged training of frozen heads and an encoder can separate class from domain information. It does so on flat CLIP and DINOv2 image models with tens to hundreds of classes, not on tens of thousands of extremely sparse labels.

The second group motivates the gap. PLM-ICD chunks notes, pools the chunk outputs, and applies label-wise attention, reporting 59.8% micro-F1 on MIMIC-III. Its authors stress how fragile the joint fine-tuning is. JointLAAT conditions leaf codes on parent categories and gains 0.8% macro-F1 on MIMIC-III-full (p < 0.01). It does this end to end, over a flat BiLSTM. Ji et al.'s BERT-hier study finds that a tuned CNN beats hierarchical BERT on frequent codes (macro AUC 90.8). It leaves open whether hierarchical Transformers lose because attention spreads too thinly across chunks or because joint training reshapes the representation. None of these papers tests across datasets. We reuse PLM-ICD's architecture and JointLAAT's parent hierarchy, and change only how and when each part is trained.

Research Program

The architecture is held fixed at PLM-ICD's design: a RoBERTa-PM chunk encoder, chunk outputs concatenated at the token level, and label-wise attention. This means every difference we report comes from training, not capacity.

Stage 1. We train the full network end to end on a reduced target set. This set contains every three-character ICD parent category and every leaf code with at least $k$ training positives, with $k$ swept over 10, 50 and 100. Parents are dense even when their children are not, so the encoder learns from labels that carry real signal rather than from one-off examples.

Stage 2. We freeze the chunk encoder. Then, in two variants, we either also freeze the document-level attention or let it adapt. In both variants we fit label queries for all leaf codes. A label query is the vector each code uses to attend over the note. Each leaf code $c$ with parent $p(c)$ gets a query $q_c = q_{p(c)} + \delta_c$, where $q_{p(c)}$ is the parent query carried over from stage 1 and $\delta_c$ is a learned residual. We train with

$$ \mathcal{L} = \sum_{c} \mathrm{BCE}_c + \lambda \sum_{c} \frac{1}{n_c} \lVert \delta_c \rVert^2 , $$

where $\mathrm{BCE}_c$ is the binary cross-entropy for code $c$, $n_c$ is its number of training positives, and $\lambda$ sets the overall strength. The $1/n_c$ factor shrinks a rare code toward its parent and lets a frequent code move freely. This is the same logic as empirical-Bayes pooling: the less evidence a code has, the more it borrows from its group.

Stage 3. We fit a per-code bias and decision threshold on the validation split under the natural code prevalence. This mirrors the second phase of the brain-lesion recipe.

Analysis. Alongside accuracy, we measure how strongly each tier's representation encodes the data source. We train a linear probe to predict the source (MIMIC-III versus later MIMIC-IV) from frozen chunk-level and document-level representations, under each training scheme. If end-to-end training puts more source information into the chunk tier than parent-supervised stage 1 does, that supports the warping account directly, independent of F1.

Baselines

Naive reference. The do-nothing reference is PLM-ICD trained end to end on all codes with default thresholds, reproduced with the authors' recipe. The paper reports 59.8% micro-F1 on MIMIC-III, and our reproduction must land near it before any comparison counts. The Mimic-IV-ICD benchmark gives the within-dataset reference for the long tail: 5.42% macro-F1 on MIMIC-IV-ICD10.

Strongest end-to-end competitor. This is PLM-ICD with JointLAAT-style parent conditioning, trained jointly, plus per-code threshold tuning on validation. It has the same ontology information and the same calibration step as our method, so it isolates the effect of staging itself. Beating it means a higher macro-F1 in the bucket of codes with at most 10 training positives, together with a smaller cross-dataset drop in that bucket, with no loss of more than half a point of micro-F1 on frequent codes.

Key ablation. This is classifier re-training on an end-to-end backbone. We take the jointly trained PLM-ICD, freeze it, and re-fit all label queries and thresholds with the same shrinkage loss. It differs from our method only in how the backbone was trained. It is the decisive test of whether the representation matters at all.

Further references. A tuned CNN (MultiResCNN-style) is included because Ji et al. found it hard to beat on frequent codes. Synthetic Clinical Notes for Rare ICD Codes reports that adding 90,000 generated notes lifts GKI-ICD macro-F1 from 12.3 to 13.2 on MIMIC-III. We cite this as an external within-dataset reference rather than re-running the generation pipeline, and say so where we compare. MSMN is described by Mimic-IV-ICD as the strongest rare-code model, but no reported result was found on Lacuna for MSMN's bucket-level scores. We will run it from its public recipe.

Fairness controls. All PLM-based methods share the backbone, chunk length and maximum note length. All get the same total GPU-hour budget, with stage 1 and stage 2 together counted against it. All get the same random search of 20 hyperparameter settings, chosen on the source validation split only, and all are run with three seeds.

Why not just…?

Why not just re-fit the classifiers and thresholds after ordinary end-to-end training? This is the real threat, and it is built in as the key ablation. Classifier re-fitting is likely to recover much of the rare-code gap within MIMIC-III or MIMIC-IV. If it matches our method out of distribution as well, the backbone-warping hypothesis is wrong for clinical text. We would then report post-hoc re-fitting as the recommendation and drop the staged pipeline.

Why not just reweight the loss or resample rare codes during joint training? The Danish infection study (Bias in Danish Medical Notes) found that weighted resampling inside a jointly trained hierarchical BERT model left error-rate gaps between cancer types largely intact. Oversampling a code with three examples mostly teaches the model those three notes. We nonetheless run focal and inverse-frequency-weighted versions of the strongest competitor, because their effect on cross-dataset rare-code transfer has not been measured.

Why not just feed the whole note to a long-context language model? On Context Utilization in Summarization with Large Language Models shows that such models favor the start and end of their input. On MiddleSum, Mistral-7B reached 7.16 ROUGE-2 with standard inference versus 10.15 with incremental, chunked processing. The evidence comes from summarization, not coding, and producing tens of thousands of calibrated per-code scores from a generative model is a separate engineering problem. We therefore treat this as out of scope rather than refuted.

Why not just use a CNN, since it already wins on frequent codes? In cross-hospital de-identification, a CNN fell from 98.2% to 54.2% F1 while a pretrained model fell from 99.7% to 78.5% (Revisiting De-Identification of Electronic Medical Records). That is a different task, so we include the CNN in the cross-dataset evaluation rather than assume it fails.

Proposed Experiments

Datasets. We use MIMIC-III-full (ICD-9, 8,921 codes) and MIMIC-IV-ICD9 from the Mimic-IV-ICD benchmark.

Cross-dataset setting. Models are trained on MIMIC-III and tested on MIMIC-IV-ICD9 admissions that fall after the MIMIC-III collection period. This removes overlapping stays. The test shift comes from changes in documentation and coding practice within one institution. It is a weaker shift than moving to a new hospital, and we state this plainly. No reported result was found on Lacuna for any ICD coder evaluated this way.

Second shift and secondary analysis. We run the reverse direction, MIMIC-IV to held-out MIMIC-III. As an exploratory step, MIMIC-IV-ICD10 is scored at the parent-category level only. This follows RuCCoD's observation of a 15% gap between exact-code and parent-group matching (RuCCoD).

Metrics. We report micro- and macro-F1, and F1 within training-frequency buckets: at most 3, 4 to 10, 11 to 100, and more than 100 positives. The central quantity is the within-to-cross-dataset drop per bucket. Macro AUC is reported but not used for decisions, given how far it diverges from macro-F1 on this benchmark.

Decisive checks. There are three, each answering one part of the core claim.

First, does staging matter out of distribution? Our method must cut the cross-dataset drop in the at-most-10 bucket relative to classifier re-training, by more than the spread across seeds.

Second, which tier matters? Freezing only the chunk tier should match or beat freezing both tiers on frequent codes. This would show that partial freezing avoids the cap on accuracy that full freezing imposes when labeled data is plentiful.

Third, is the mechanism the one we claim? The probe for data source should be markedly less accurate on chunk representations from parent-supervised stage 1 than on those from end-to-end training.

If the first check fails while the third succeeds, the backbone is affected but the effect does not reach the classifiers. That result would point future long-tail work at the classifier stage, which is cheaper to fix.