Automated medical coding assigns billing and diagnosis codes (ICD codes, the International Classification of Diseases) to hospital discharge notes. Thousands of codes are possible, and most appear only a handful of times. Every published model, including those built on biomedical language models, still does badly on these rare codes. The usual response is to treat this as a fine-tuning problem and add class reweighting, resampling, or special losses. This note proposes a different diagnosis. Rare codes tend to be described by rare clinical words. Language model pre-training is known to get worse at rare words as vocabularies and models grow. So part of the rare-code failure may already be present in the encoder before any coding data is seen. Fine-tuning tricks cannot repair that.
We propose to test this directly on reproducible MIMIC coding benchmarks and to intervene where the damage happens. The intervention is continued pre-training that shifts learning toward rare clinical words, using only publicly shareable text so anyone can reproduce the encoder. We treat overplotting in a narrow way: as a lens for measuring how the documents of rare codes get buried among majority-code documents in the model's representation space. Overplotting is the visualization problem where many points land on the same pixel and minority points disappear. If the diagnosis holds, it moves effort on long-tail clinical coding from the classifier head to the pre-training objective.
Core Claim
Hold fixed how often a code appears in the coding training set. Codes whose descriptive vocabulary the encoder predicts poorly during pre-training will have lower per-code F1 after fine-tuning. Their documents will also sit more crowded among other codes' documents in the fine-tuned representation space. A frequency-aware continued pre-training objective on public biomedical text should shrink both effects. It should improve rare-code Macro F1 beyond what the best fine-tuning-stage imbalance remedies achieve alone, and the two should combine rather than substitute for each other.
The claim rests on three findings in the file. Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training shows that on general web text, growing the vocabulary from 24K to 196K lowers overall loss. The gain comes almost entirely from the top 2,500 words, while loss on the rarest 20,000 words rises from 11.183 to 13.399 nats. The authors test only general-domain corpora and explicitly flag tasks where rare words matter as untested, so this paper motivates the proposal. Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study re-implements six coding models and corrects the Macro F1 formula. It builds a stratified "MIMIC-III clean" split and concludes that the long tail, not document length, drives failure. This paper motivates the proposal and also constrains how it must be evaluated. PLM-ICD adapts biomedical encoders with label-wise attention and segment pooling, and it supplies our base architecture. None of these papers connects a code's tail performance to how well the encoder learned that code's words.
Research Program
The first component is a diagnostic study. For each ICD code we take the words of its official description plus the words most associated with it in training notes. We call these the code's "descriptor terms." We then measure how well a given biomedical encoder predicts these terms under masked-word prediction on held-out public biomedical text. That gives each code a pre-training loss score.
Next we fine-tune PLM-ICD on the MIMIC-III clean split and on the MIMIC-IV ICD-9 and ICD-10 splits, using the corrected evaluation protocol from the replicability study. We regress per-code F1 on the pre-training loss score, controlling for the log frequency of the code in the training labels. The control matters because rare codes will trivially have both rare words and low F1. The test is whether the loss score carries information beyond code frequency.
We also compute a "crowding index" for each code, which is our overplotting measure moved from pixels to representations. For each test document carrying a code, we take the label-attention representation for that code and count the fraction of its $k$ nearest neighbors, in the full-dimensional space, that also carry the code. We deliberately avoid a 2D projection. WizMap partitions 2D projections with quadtrees. Dual Space Coupling Model Guided Overlap-Free Scatterplot relaxes point positions to remove overlap. Both operate only on 2D layout coordinates and ignore class labels, so a dense majority cluster can visually swallow a minority one. Our index measures the burial before any projection, and we then use it to choose where a visual display should zoom in. This is a diagnostic, not a new layout algorithm.
The second component is the intervention: frequency-aware continued pre-training. We start from a public biomedical encoder and continue masked-word pre-training on public text only. The text is PubMed abstracts together with the 158k synthetic clinical notes and instructions released by Asclepius. Asclepius rewrote public case reports into discharge-summary style and matched a MIMIC-trained twin in blinded clinician review ($p = 0.18$). That result supplies our route to a shareable clinical corpus, although it was validated on generation tasks, not coding.
The objective raises the weight on masked positions holding rare words:
$$ \mathcal{L} = -\sum_{i \in M} w(t_i)\,\log p_\theta(t_i \mid x_{\setminus M}), \qquad w(t) = \left(\frac{f_{\text{med}}}{f(t)}\right)^{\beta}, $$
In this objective, $M$ is the set of masked positions, $t_i$ is the word at position $i$, and $x_{\setminus M}$ is the input with those positions hidden. The weight $w(t)$ depends on $f(t)$, the corpus frequency of word $t$, and $f_{\text{med}}$, the median word frequency. The weight is clipped to $[1, w_{\max}]$ so common words are never down-weighted below standard training, and $\beta \geq 0$ sets how strongly the objective tilts toward the tail ($\beta = 0$ recovers ordinary masked-word training). We also raise the masking probability of rare words by the same factor, so they are hidden and predicted more often.
The tokenizer is kept fixed. Chung and Kim's result suggests that simply adding domain vocabulary can deepen the rare-word tax rather than relieve it, and a fixed tokenizer keeps the comparison clean.
The third component is a reproducibility audit. We build a second encoder by continued pre-training on real MIMIC notes, which requires credentialed access under a Data Use Agreement (MIMIC-III). This quantifies how much rare-code performance a fully public pipeline gives up. All public-corpus encoders, split files, and threshold-tuning code will be released.
Baselines
The naive reference point is PLM-ICD fine-tuned from its public biomedical encoder with standard binary cross-entropy and a single global threshold tuned on validation data. The reported number for this system is micro-F1 of 59.8% on MIMIC-III (PLM-ICD), obtained on the original split and evaluation. The replicability study reports that correcting Macro F1 roughly doubles prior Macro F1 values. However, the file contains no Macro F1 value on MIMIC-III clean or on MIMIC-IV: no reported result found on Lacuna for PLM-ICD Macro F1 on MIMIC-III clean or MIMIC-IV. We will therefore re-run this baseline ourselves under the corrected protocol and report it alongside our systems.
The strongest competitor on this question is a set of fine-tuning-stage imbalance remedies stacked on the same PLM-ICD model. The first is the careful-tuning recipe from Simplifying Neural Network Training Under Class Imbalance: small batches, class-conditional label smoothing, and sharpness-aware minimization. It beat specialized imbalance losses on image and tabular data and cut error on the 20 smallest CIFAR-100 classes by 5.4% to 6.8%. The second is the closed-form per-class reweighting from Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem, adapted to multi-label training by reweighting per code. It reported a 6.26% to 7.08% Top-1 gain on CIFAR-100-LT. The third is per-code threshold tuning, which the replicability study found closes large gaps between architectures.
These remedies are the strongest competitors because they are the accepted way to fix long tails, and they need no new pre-training. Each gets the same hyperparameter search budget as our method, measured in trials and GPU-hours, on the same splits.
The key ablations isolate the pre-training ingredient. One is continued pre-training on the identical public corpus with ordinary masked-word loss ($\beta = 0$) and identical compute, which separates "more in-domain text" from "tilting toward rare words." Another is continued pre-training that oversamples documents containing rare descriptor terms, which separates reweighting inside the loss from resampling the data.
Beating the strongest competitor means one of two outcomes on both MIMIC-III clean and MIMIC-IV ICD-10, averaged over at least three seeds. The first is that our encoder plus plain fine-tuning exceeds PLM-ICD plus the best fine-tuning remedy on Macro F1 for the rarest frequency tier of codes. The second is that our encoder plus the best remedy exceeds that remedy alone on tail Macro F1 by more than the seed standard deviation, without losing more than one point of micro-F1.
Why not just…?
Why not just reweight the loss during fine-tuning? This is the main threat, and the experiment is built around it. If per-code reweighting and careful tuning recover rare-code F1 as fully as our encoder does, then the problem lives in the classifier, and the claim fails. We expect it will not fully recover. With a handful of positive examples, reweighting only amplifies gradients through representations of the code's words, and if those representations are poor, amplifying them does not help. The factorial design settles this: we cross the two encoders with the two fine-tuning regimes. If the benefit of our encoder vanishes once fine-tuning is reweighted, we report that the two are substitutes.
Why not just add medical words to the tokenizer? Domain-specific vocabularies are the common fix. However, Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training found that larger vocabularies sharpen frequency imbalance and raise loss on rare words. A domain vocabulary may therefore make each rare medical term a rare token that the model predicts even worse. We include one run with an expanded clinical vocabulary under standard masked-word training as a secondary comparison. We do not assume which way it will go.
Why not just pre-train on real MIMIC notes, or simply on more domain text? MonoCoder and A Deep Fusion Model for Domain Adaptation in Phrase-based MT show that focused domain data helps. The $\beta = 0$ ablation tests whether in-domain text alone already closes the gap, holding compute equal. The MIMIC-note encoder tests what the public-only constraint costs. If real notes with plain training beat public text with reweighting, the reproducibility trade-off becomes the headline result.
Why not just visualize the embeddings with an existing tool? WizMap and overlap-free layouts can show where points pile up. However, they reason in 2D and ignore labels, so their picture of which codes are buried depends on the projection. We therefore use the full-dimensional crowding index as the quantitative measure and treat visualization only as a way to inspect cases the index flags.
Proposed Experiments and Decisive Checks
All evaluation uses the MIMIC-III clean split and the MIMIC-IV ICD-9 and ICD-10 splits from the replicability study. Macro F1 is computed as the arithmetic mean of per-code F1 over codes present in the test set, and thresholds are tuned on validation data. We report micro-F1 and Macro F1 overall and within code-frequency tiers: codes with fewer than 10 training examples, 10 to 100, and more than 100. The main design is the 2x2 factorial of encoder (standard or frequency-aware continued pre-training) by fine-tuning (plain or best remedy), plus the ablations above, each run with three seeds.
The first decisive check is the diagnostic regression. If the pre-training loss score adds no explanatory power for per-code F1 beyond code frequency, the mechanism is wrong and we stop before the intervention.
The second check is mediation through representations. The intervention should lower the loss score on descriptor terms and raise the crowding index for tail codes. Tail F1 gains should track those two changes code by code. A gain without a representation change would point to some other cause.
The third check tests generality. We repeat the diagnostic on a second biomedical encoder, and we also inspect generation benchmarks such as MeQSum. There BioBART reached Rouge-L 53.15 against 51.05 for general-domain BART-Large (BioBART). We use it only to confirm that the reweighted encoder does not degrade general biomedical ability; we are not claiming gains there.
References
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- PLM-ICD: Automatic ICD Coding with Pretrained Language Models
- WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings
- Dual Space Coupling Model Guided Overlap-Free Scatterplot
- Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes
- MIMIC-III, a freely accessible critical care database
- Simplifying Neural Network Training Under Class Imbalance
- Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of View
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model