TL;DR
Researchers have developed a hybrid pipeline for ICD-10 medical coding that uses a supervised model to generate candidate codes and a "gate" to selectively call a Large Language Model (LLM) only when necessary. This approach achieves 83.19% accuracy on Russian clinical text—outperforming full LLM invocation—while reducing expensive LLM calls by 68%.
What Problem Does This Paper Address?
Assigning ICD-10 codes (the international standard for reporting diseases) to clinical notes is a cornerstone of modern healthcare, affecting everything from billing to epidemiology. However, doing this automatically is notoriously difficult. Clinical text is often short, riddled with abbreviations, and contains typos. Furthermore, the ICD-10 system is hierarchical and suffers from extreme "class imbalance"—a few common conditions appear everywhere, while thousands of others are incredibly rare.
While LLMs are powerful tools for text normalization, calling a massive model for every single diagnosis is prohibitively expensive and slow for real-world deployment. The authors sought a way to maintain high accuracy without the high "inference tax" of unrestricted LLM use.
Methods
The authors propose a multi-stage pipeline designed for efficiency (visualized in Figure 2):
- Supervised Candidate Generation: The system first uses ModernBERT to create semantic embeddings of the text. It then employs a two-level hierarchical classifier: a Random Forest predicts the broad ICD-10 letter (e.g., "J" for respiratory diseases), followed by a Histogram-based Gradient Boosting (HGB) model that predicts the specific "parent code" (e.g., "J02"). This stage produces a Top-10 candidate list.
- Selective LLM Invocation (The "Gate"): Instead of sending every case to an LLM, a learned "Pre-LLM gate" predicts whether the supervised model is likely to be wrong and if an LLM call is worthwhile.
- Reranking: If the gate is triggered, an LLM (specifically Qwen3.5-122B-A10B or a GPT-based baseline) reranks the Top-10 candidates. If not, a lightweight supervised ranker handles the task.
- Fusion: A final selector combines these paths to produce the definitive parent code.
Datasets
The study uses a dataset of short clinical diagnosis formulations in Russian.
- Reference Data: 9,930 ICD-10 diagnosis-code pairs from the Russian Ministry of Health.
- Evaluation Data: A final split of 8,552 training rows and 1,511 "locked-test" rows. The authors used a strict "leakage-safe" protocol, ensuring no shared source files between training and testing sets to prevent the model from simply memorizing specific doctors' writing styles.
Key Insights and Ideas
- LLMs as Rerankers, Not Generators: The most effective role for an LLM in this pipeline is not to "guess" the code from scratch, but to act as a sophisticated judge choosing between a few high-probability candidates generated by a faster model.
- The Power of the Gate: By identifying "uncertain" examples, the gate allows the system to spend its "compute budget" only where it adds the most value.
- Candidate Coverage is King: The supervised generator achieved a 93.05% Top-10 coverage. This high "recall" is what makes downstream LLM reranking viable; if the correct code isn't in the Top-10, the LLM (in this constrained setting) can't find it.
- Imbalance Persists: Even with LLMs, rare classes remain a challenge. As shown in Figure 1, the data is heavily skewed toward a few chapters (like J, K, M, and N), and performance on rare letters is significantly lower.
Key Results
- Accuracy Boost: The final "Pre-LLM fusion" system reached 83.19% Top-1 accuracy on the locked test set.
- Efficiency: This was achieved while calling the LLM for only 31.97% of cases.
- Outperforming the "Gold Standard": Surprisingly, the hybrid system outperformed a direct zero-shot GPT-5.6-Sol baseline (81.01% accuracy) despite the latter using LLM calls 100% of the time.
- Error Reduction: Compared to the supervised-only baseline (74.32%), the hybrid approach reduced the error rate by 34.5%.
Limitations and Future Work
- Rare Class Performance: While the system is more accurate overall, rare classes still suffer from lower recall and F1-scores. Errors in rare classes are often "redirected" by the model to more frequent, similar-looking categories.
- Open-Set Challenges: The current pipeline assumes the correct code is within the supervised model's candidate list. It does not yet explicitly handle "unseen" parent codes that were not present in the training data.
- Dataset Specificity: The results are validated on a specific Russian clinical dataset; the authors note that different languages or medical contexts might require retuning the balance between the gate and the reranker.
Future work will likely focus on "rare-class-aware" training strategies, such as hierarchical losses or specialized resampling, to close the gap for less common medical conditions.
Summary updated