A one-billion-parameter language model fine-tuned on medical codes can now assign the right disease family to a hospital discharge note most of the time, but it often picks the wrong specific code inside that family. The ICD-10 diagnostic taxonomy has more than 74,000 codes. On real MIMIC-IV notes, the specialized Llama-3.2-1B of Enhancing medical coding efficiency through domain-specific fine-tuned large language models gets the three-character category right 87.16% of the time and the exact code right only 69.20% of the time. Roughly one correct-family prediction in five therefore lands on a sibling: the wrong laterality, the wrong episode of care, a code that is too broad or too specific. These are the errors that matter for billing and audit, and they are a domain instance of a general problem in many-class classification. As the label space grows, mistakes stop being spread out and concentrate among a small number of near-duplicate classes.
We propose to fix this by spending synthetic-data budget only where the student is confused, and only on contrasts that teach the missing distinction. During training we track how much probability the student assigns to each sibling of the gold code, giving a running "soft" confusion score per sibling pair. We then ask a teacher model to write minimal pairs: two clinical notes that are identical except for the detail that separates the two confused codes. The two-sentence version is this. Uniform synthetic augmentation wastes most of its examples on distinctions the model already makes. Confusion-targeted minimal pairs restricted to the taxonomy's sibling sets should close the gap between category accuracy and exact-code accuracy at the same data budget.
Core Claim
We claim that the residual error of compact specialized coders is a sibling-discrimination problem, not a hierarchy-awareness problem. If that is right, the natural import from the many-class vision literature, which is tree-structured regularization, is the wrong tool. Pulling siblings together in representation space is what A Closer Look at Few-Shot Classification with Many Novel Classes does with Cophenetic-correlation and hierarchy-triplet losses, and what Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy does with multi-level superclass prediction. Both help when errors land in distant branches. Hou et al.'s error profile shows the opposite: the model is already in the right branch and fails at the last step. A generative coder that writes the code character by character already decodes the hierarchy prefix-first. We therefore predict that a tree loss will leave exact match unchanged or lower it, and that confusion-targeted contrast will raise it. That prediction is falsifiable, and it is the sharpest thing this note claims.
The mechanism takes its ideas from two papers. From Confusable Learning for Large-class Few-Shot Classification it takes the soft confusion matrix, which tracks predicted probabilities rather than hard error counts so that it registers near-misses on correct predictions, and the practice of building training tasks around a class's worst distractors. From Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale (PGKD) it takes the loop in which student errors are reported to a teacher that writes targeted training examples. The proposal differs from each in a specific way. Confusable Learning samples episodes from existing images and never creates new data. PGKD triggers only on confident hard misclassifications over a flat 335-class space, and it generates independent examples rather than contrasts. Neither scales its confusion statistics to tens of thousands of classes.
Research Program
The confusion statistic is kept tractable by restricting it to the taxonomy. A full 74,000 by 74,000 matrix is infeasible and is exactly the meta-testing overhead that Lin et al. flag. For a note $x$ with gold code $i$, let $S(i)$ be the codes sharing $i$'s three-character category. We score each candidate $j \in S(i)$ by teacher-forced likelihood under the student, renormalize over $S(i)$ to get $p(j \mid x)$, and maintain
$$C_{ij} = \frac{1}{|W_i|}\sum_{x \in W_i} p(j \mid x), \quad j \in S(i),\ j \neq i,$$
where $W_i$ is a sliding window of recent training notes whose gold code is $i$. Sibling sets typically hold tens of codes, so the matrix is block-sparse and costs one batched scoring pass per note. Pairs are selected in proportion to $C_{ij}$ weighted by code frequency in the training notes. Rare codes are included through their official descriptions, so the tail is not starved.
For each selected pair, the teacher receives both official ICD-10 descriptions and one seed note. It is asked to name the clinical detail that separates the two codes and to write two versions of the note that differ only in that detail. Each pair is checked automatically before use. A separate strong model, unaware of which version is which, must assign each note to the correct code. The two notes must also overlap heavily at the token level, so that the pair really is minimal. The student, starting from Hou et al.'s two-phase recipe, is trained on these pairs with ordinary next-token loss. One variant adds a margin term that pushes $\log p(i \mid x)$ above $\log p(j \mid x)$ on the partner note.
Teacher dependency, an open problem raised by both Fu et al. and Singhi et al., is tested directly. We run the loop with a frontier teacher and with an open-weight teacher, and report how much of the gain survives the switch.
A secondary study on AMZN Reviews (335 classes, the PGKD benchmark) checks whether the method transfers outside medicine. There is no given taxonomy there, so sibling sets are formed by clustering label embeddings. This tests whether the method needs a clean hierarchy, a dependence Li et al. identify as fragile.
Baselines
The naive reference point is the Phase-2 model of Hou et al.: Llama-3.2-1B fine-tuned on the 74,260 code–description pairs and then on uniformly sampled synthetic clinical variations. Their reported MIMIC-IV result is 69.20% exact match and 87.16% category match. We cannot assume their exact data split is available, so we re-run their recipe on our MIMIC-IV evaluation set and treat their figure as a sanity anchor rather than the comparison number.
The strongest competitor on this question is uniform augmentation at matched budget. This is Hou et al.'s Phase-2 generator extended until it produces exactly as many synthetic notes, and the same teacher tokens, as our method. This is the real threat. If our gain comes merely from more data, this baseline will match it. Beating it means a statistically reliable gain in exact match with category match held roughly fixed. That shows up as a higher within-category resolution rate, which is exact match divided by category match, about 0.79 for the reported baseline.
The second competitor is PGKD transplanted to this setting. It uses the same teacher and budget, but generation is triggered by confident hard errors and produces independent examples. Its reported result on its own benchmark is accuracy rising from about 32% to over 44% on AMZN Reviews. On that benchmark our method must beat a re-run of PGKD at matched teacher calls.
The third competitor is the obvious transfer from vision: the Phase-2 model plus a hierarchy loss, either CPCC regularization or hierarchy triplet loss as in Lin et al.
Three ablations separate the ingredients. Minimal pairs on randomly chosen sibling pairs isolate confusion targeting. Independent examples for confused codes, rather than pairs, isolate the minimal-pair format. Hard confusion counts in place of $C_{ij}$ isolate the soft signal. Every arm uses the same student, the same number of optimizer steps, and the same hyperparameter search budget, with three seeds.
Why not just…?
Why not just generate more uniform synthetic notes? Hou et al. already showed that uniform variation training moves multi-condition accuracy from roughly 4–10% to 94–98%, so more of it is a plausible fix. Our expectation is that its returns at the sibling level are low, because most uniformly sampled examples exercise distinctions the student already makes. The matched-budget baseline settles this directly. If it matches us, the proposal reduces to "more data," and we will report that.
Why not just add a hierarchy loss, as the many-class vision work does? Tree losses from Lin et al. and Li et al. reward placing an example near its family, and our student already does that 87% of the time. Shrinking distances between siblings could make sibling confusion worse. We test this explicitly and expect the result to be negative. A positive result would contradict our core diagnosis.
Why not just retrieve the sibling codes and let the model choose among their descriptions at inference time? This shortlist-then-rerank approach is cheap and is a genuine alternative for the final decision step. It does not, however, teach the model which note detail matters. We include it as an inference-time variant applied both to the baseline and to our model. If it gives the baseline the same gain our training gives, the training contribution is moot. If the gains stack, the two are complementary.
Why not just prompt a frontier model directly? Hou et al. report under 1% zero-shot exact match on ICD-10, and Singhi et al. measure direct LLM inference as 25 times more expensive than a distilled student. We do not re-run this baseline.
Proposed Experiments
The primary evaluation is MIMIC-IV discharge summaries mapped to ICD-10. We report exact match, category match, within-category resolution rate, and exact match on codes bucketed by training frequency. The tail bucket matters because Hou et al. left rare codes untested. Generality is checked on held-out ICD-10-CM description pairs, to confirm the Phase-1 "code dictionary" is not eroded, and on AMZN Reviews, where the comparison point is PGKD's reported >44%.
Three outcomes decide the proposal. First, confusion-targeted minimal pairs must beat matched-budget uniform augmentation on within-category resolution, and the measured $C_{ij}$ for targeted pairs must fall over training while untargeted pairs do not. Second, the hierarchy-loss arm must fail to improve exact match. Third, the randomly-paired and independent-example ablations must each recover only part of the gain.
If the first check fails, the idea is wrong. If the second fails, our diagnosis of the errors is wrong. The note's value lies in finding out which.
References
- Enhancing medical coding efficiency through domain-specific fine-tuned large language models
- A Closer Look at Few-Shot Classification with Many Novel Classes
- Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy
- Confusable Learning for Large-Class Few-Shot Classification
- Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale