Hierarchy-Anchored Low-Rank Label Vectors for Rare ICD Codes

Automated medical coding assigns billing and diagnosis codes (ICD codes) to hospital discharge notes. It is a classification problem with tens of thousands of labels, and most of those labels are almost never seen. On the MIMIC-IV ICD-10 benchmark, over half of 26,096 codes appear three times or fewer. Strong models reach macro AUC up to 97.07% but macro F1 as low as 5.42% (Mimic-IV-ICD). A careful replication study found that the long tail of rare codes, not document length, is the main cause of errors (Automated Medical Coding on MIMIC-III and MIMIC-IV). Yet every one of these models gives each code the same large learned vector, typically 768 numbers. That vector tells the model where to look in the note and how to score it. A code with two positive notes therefore has hundreds of free parameters fit to two examples.

We propose to stop giving every code the same budget. Each code's vector is built as the sum of its ancestors' vectors in the ICD hierarchy, plus a small correction of its own. That correction lives in a low-dimensional subspace shared by all codes in the same ICD chapter, and its size grows with how often the code is observed. A rare code can therefore only move away from its parent along a few directions that its data-rich siblings have already found useful. Recommender systems and knowledge graphs have shown that shrinking embeddings for rare entities helps. The new questions here are whether that holds for output labels in clinical coding, and whether the hard, shared-subspace form of the constraint beats the obvious soft alternative of pulling rare codes toward their parents with a penalty.

Core Claim

Prior work on variable embedding sizes allocates dimensions by frequency alone, on flat vocabularies of input entities. CustomizE picks per-entity sizes from occurrence counts using bilevel search on general knowledge graphs (Should We Use a Fixed Embedding Size?). SSEDS prunes dimensions per feature field using a single gradient-based saliency pass (Single-shot Embedding Dimension Search). OptEmbed prunes feature values and searches field sizes with a weight-sharing supernet (OptEmbed). These papers supply the method: separating how many numbers are stored from the size used in computation, and nesting smaller sizes inside larger ones.

The clinical coding work motivates the proposal. ICDBigBird models relations between codes with a graph network and a margin loss, but it gives every code the same size and evaluates only the top 50 codes. The MIMIC-IV benchmarks keep uniform label vectors throughout.

Our claim is that, for extremely rare codes, the form of the regularizer matters. We compare a hard rank limit on a code's deviation from its ancestors, taken inside a chapter-shared learned subspace, against an equal-strength isotropic shrinkage toward the same ancestors. The shrinkage treats all directions of deviation equally. We predict the rank-limited version will raise macro F1 on codes with three or fewer training positives, and will do so with a label layer several times smaller. If the soft version matches it, the finding is still useful: the gain comes from anchoring to the hierarchy, not from the embedding dimension. The experiments are designed to tell these two outcomes apart.

Mechanism

Let $u_c \in \mathbb{R}^{D}$ be the vector for code $c$, where $D$ is the encoder width. In models like LAAT and PLM-ICD, $u_c$ plays two roles: it is the per-label attention query and the per-label output weight. We parameterize both roles the same way:

$$u_c = \sum_{a \in \mathrm{anc}(c)} v_a \;+\; P_{g(c)}[:, 1{:}r_c]\, z_c .$$

The terms are defined as follows. $\mathrm{anc}(c)$ is the chain of ICD-10 ancestors of $c$, from chapter to block to three-character category. Each $v_a \in \mathbb{R}^{D}$ is a full-size vector for an ancestor. Ancestors pool the positives of all their descendants, so they are well estimated. $P_{g} \in \mathbb{R}^{D \times R}$ is a projection shared by all codes in chapter $g$. The code uses only its first $r_c$ columns, in the same nested style as OptEmbed's ordinal sharing. Finally, $z_c \in \mathbb{R}^{r_c}$ is the code's own small vector.

The default rule sets the rank as $r_c = \min(R, \lceil \kappa \log(1+n_c) \rceil)$, where $n_c$ is the number of training notes containing $c$. We use a logarithm because it is a simple monotone rule that gives head codes large ranks without letting the parameter count grow linearly with frequency. We will also test a learned variant. It applies SSEDS-style first-order saliency at the level of individual codes rather than fields: train once, score each column of each code's correction, then keep the highest-scoring columns under a global parameter budget. This avoids the bilevel search that the CustomizE authors list as their scaling bottleneck.

The rank can be zero. A code with zero rank is its parent plus a per-code bias. This also gives a principled score for codes absent from training. Where codes have official definitions, $z_c$ can additionally be initialized from definition text, following ICDBigBird's finding that definition initialization beats random initialization.

Research Program

We will build the parameterization as a replacement label layer for two encoders. LAAT is a BiLSTM with label attention and is cheap enough for many runs. PLM-ICD is a clinical RoBERTa with label attention and is the strongest pretrained baseline benchmarked on MIMIC-IV. The encoder, input truncation, optimizer and training schedule stay fixed, so any change comes from the label layer.

The analysis has three parts. The first is per-tier results, with codes grouped by training frequency: three or fewer, 4 to 10, 11 to 100, and over 100. The second is a memorization diagnostic: the gap between training and validation F1 in each tier, which directly tests the claim that full-size rare vectors overfit. The third is a subspace inspection: we check whether rare codes' corrections load onto the directions that dominate frequent siblings' corrections, which is the proposed mechanism.

Baselines

The naive reference point is the uniform model. This is PLM-ICD, and separately LAAT, with a full $D$-dimensional vector per code, trained with binary cross-entropy. Per-code decision thresholds are tuned on validation data exactly as in the replication study. That study showed threshold tuning alone roughly doubled macro F1 and erased many apparent differences between architectures, so every model here gets the same tuning. The reference number for the target benchmark is the macro F1 of 5.42% and macro AUC of up to 97.07% on MIMIC-IV-ICD10 reported by Mimic-IV-ICD. No per-model breakdown (PLM-ICD versus MSMN) was found on Lacuna. No reported MIMIC-IV figure from the replication study was found on Lacuna either. We will therefore rerun all baselines ourselves on identical stratified splits, using the correct arithmetic mean of per-class F1.

The strongest existing tail method is MSMN. According to the MIMIC-IV-ICD benchmark, it outperforms PLM-ICD on rare codes by matching note text to code synonyms. We run it from its released configuration on our splits. Beating it means higher macro F1 in the three-or-fewer tier without losing to PLM-ICD on the head.

We also include PLM-ICD with the margin loss used in ICDBigBird (LDAM, a loss that enforces larger decision margins for rarer classes), because this is the standard objective-level fix for imbalance.

Four ablations isolate the ingredients. The first is a flat frequency-bucketed allocation with no hierarchy, in the spirit of CustomizE-rule: each code gets its own rank-$r_c$ vector through a single global projection. This tests whether frequency-based sizing alone explains the gain. The second is a uniform low-rank label matrix at a matched parameter count. This is what field-level pruning in the style of SSEDS reduces to when all codes form one field, and it tests whether simply having fewer parameters is enough. The third is the key threat: the hierarchy sum with a full-size per-code correction and an L2 penalty toward the ancestors, scaled as $\lambda/n_c$. Its strength is tuned over the same number of trials as our $\kappa$. The fourth replaces the true ICD tree with a random tree that has the same branching profile. This tests whether the ontology matters or any grouping works.

All methods get the same encoder checkpoints, the same number of training epochs and the same hyperparameter-search budget of equal trial count on validation macro F1. Each is run with three seeds. We report parameter counts for every label layer.

Why not just…?

Why not just use a reweighted or margin loss like LDAM? A loss changes how much each error costs, but it does not change how many free parameters a two-example code has. ICDBigBird's margin-loss gains were shown only on the 50 most frequent codes, and the full MIMIC-IV-ICD10 tail still sits near 5% macro F1 (Mimic-IV-ICD). Losses and parameterization are also compatible. We run LDAM both as a baseline and on top of our layer. If LDAM on uniform vectors matches our model in the three-or-fewer tier, our claim fails.

Why not just tune per-code thresholds? Threshold tuning is a strong fix and the replication study showed it matters. The same study still found all tuned models failing on rare codes. Threshold tuning moves a cutoff on scores; it cannot fix scores that are poorly ranked because the underlying vector memorized noise. We tune thresholds for every model, so any improvement we report comes on top of threshold tuning, not instead of it.

Why not just bucket codes by frequency by hand? CustomizE reports that its rule-based variant was clearly worse than data-driven selection on knowledge graphs. Frequency is also a poor proxy in coding, where a common unspecified code can be semantically vague and a rare syndrome very specific. The flat-bucket ablation measures this directly. The learned saliency variant is our answer if the log rule underperforms.

Why not just tie rare codes to their parents with a stronger penalty? This is the most serious alternative. Isotropic shrinkage toward ancestors is a sensible Bayesian prior and may capture most of the benefit. Our bet is that deviations restricted to a chapter-shared subspace, learned from frequent siblings, generalize better than deviations that are cheap in every direction. The frequency-scaled L2 ablation, tuned with equal effort, settles the question. If it ties with our model, we will report that the gain comes from hierarchy anchoring. We will then present the rank form only for its memory savings, which remain real at 26,096 codes.

Why not just initialize from code definitions, as MSMN and ICDBigBird effectively do? External text helps the tail, and we use it as an initializer where available. Initialization, however, fixes only the starting point, not the number of free parameters, so a full-size vector can still drift toward noise. Crossing definition initialization with each parameterization shows whether the two effects add up or overlap.

Why not just generate synthetic notes for rare codes? Generative augmentation struggles when a class has one to five real examples, and naive oversampling leads to memorization. Generating realistic notes for thousands of rare, co-occurring codes is also far more expensive than changing the label layer. We do not pursue it.

Proposed Experiments

The primary benchmark is MIMIC-IV-ICD10, with 26,096 codes and about 122,000 notes. We use MIMIC-IV-ICD9, with 11,331 codes, as a second label system with a different hierarchy, and MIMIC-III clean, with its stratified splits, for comparability with the replication study.

Metrics are:

  • macro F1 computed as the mean of per-code F1;
  • micro F1, macro AUC and precision at 8 and 15;
  • macro F1 within each frequency tier;
  • the train–validation F1 gap per tier;
  • label-layer parameter count.

For codes absent from training, we additionally report recall on a held-out set of rare codes whose positives are moved entirely to test. This probes the zero-rank behavior.

The decisive checks come from four comparisons. The proposal succeeds if three conditions hold together, on both encoders and both ICD versions. First, the hierarchy-anchored low-rank layer beats threshold-tuned uniform PLM-ICD and MSMN on three-or-fewer-tier macro F1, by more than the spread across seeds. Second, it does not reduce macro F1 on codes with over 100 positives. Third, it shrinks the train–validation gap in the rare tiers.

The mechanism claim holds only if the model also beats the tuned frequency-scaled L2 shrinkage at equal tuning effort, and if the random-tree control loses most of the gain. A tie with shrinkage changes the conclusion to "hierarchy, not dimension," as stated above. A random tree performing as well as the real one would mean the effect is generic regularization, and we would report that honestly.