Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation

Ilia Kopanichuk, Daniil Kartashev, Yuri Bolkonsky, Petr Anokhin, Vladimir Makharev, Iaroslav Bespalov, Dmitry V. DylovarXiv 2026 · arXiv

TL;DR

Researchers have developed a hybrid pipeline for ICD-10 medical coding that uses a supervised model to generate candidate codes and a "gate" to selectively call a Large Language Model (LLM) only when necessary. This approach achieves 83.19% accuracy on Russian clinical text—outperforming full LLM invocation—while reducing expensive LLM calls by 68%.

What Problem Does This Paper Address?

Assigning ICD-10 codes (the international standard for reporting diseases) to clinical notes is a cornerstone of modern healthcare, affecting everything from billing to epidemiology. However, doing this automatically is notoriously difficult. Clinical text is often short, riddled with abbreviations, and contains typos. Furthermore, the ICD-10 system is hierarchical and suffers from extreme "class imbalance"—a few common conditions appear everywhere, while thousands of others are incredibly rare.

While LLMs are powerful tools for text normalization, calling a massive model for every single diagnosis is prohibitively expensive and slow for real-world deployment. The authors sought a way to maintain high accuracy without the high "inference tax" of unrestricted LLM use.

Methods

The authors propose a multi-stage pipeline designed for efficiency (visualized in Figure 2):

  1. Supervised Candidate Generation: The system first uses ModernBERT to create semantic embeddings of the text. It then employs a two-level hierarchical classifier: a Random Forest predicts the broad ICD-10 letter (e.g., "J" for respiratory diseases), followed by a Histogram-based Gradient Boosting (HGB) model that predicts the specific "parent code" (e.g., "J02"). This stage produces a Top-10 candidate list.
  2. Selective LLM Invocation (The "Gate"): Instead of sending every case to an LLM, a learned "Pre-LLM gate" predicts whether the supervised model is likely to be wrong and if an LLM call is worthwhile.
  3. Reranking: If the gate is triggered, an LLM (specifically Qwen3.5-122B-A10B or a GPT-based baseline) reranks the Top-10 candidates. If not, a lightweight supervised ranker handles the task.
  4. Fusion: A final selector combines these paths to produce the definitive parent code.
Figure 2: Final call-efficient pipeline for ICD-10 parent-code prediction. The supervised candidate generator produces a shared top-10 list. A no-call candidate-ranker route and a selectively invoked LLM-reranker route then converge in the Pre-LLM fusion selector, which outputs the final parent code. Green denotes supervised or no-call processing, while blue denotes selectively invoked LLM processing.

Datasets

The study uses a dataset of short clinical diagnosis formulations in Russian.

  • Reference Data: 9,930 ICD-10 diagnosis-code pairs from the Russian Ministry of Health.
  • Evaluation Data: A final split of 8,552 training rows and 1,511 "locked-test" rows. The authors used a strict "leakage-safe" protocol, ensuring no shared source files between training and testing sets to prevent the model from simply memorizing specific doctors' writing styles.

Key Insights and Ideas

  • LLMs as Rerankers, Not Generators: The most effective role for an LLM in this pipeline is not to "guess" the code from scratch, but to act as a sophisticated judge choosing between a few high-probability candidates generated by a faster model.
  • The Power of the Gate: By identifying "uncertain" examples, the gate allows the system to spend its "compute budget" only where it adds the most value.
  • Candidate Coverage is King: The supervised generator achieved a 93.05% Top-10 coverage. This high "recall" is what makes downstream LLM reranking viable; if the correct code isn't in the Top-10, the LLM (in this constrained setting) can't find it.
  • Imbalance Persists: Even with LLMs, rare classes remain a challenge. As shown in Figure 1, the data is heavily skewed toward a few chapters (like J, K, M, and N), and performance on rare letters is significantly lower.
Figure 1: Training-set distribution by ICD-10 letter. The fre- quencies are strongly imbalanced, with letters such as J, K, M, and N dominating the dataset.

Key Results

  • Accuracy Boost: The final "Pre-LLM fusion" system reached 83.19% Top-1 accuracy on the locked test set.
  • Efficiency: This was achieved while calling the LLM for only 31.97% of cases.
  • Outperforming the "Gold Standard": Surprisingly, the hybrid system outperformed a direct zero-shot GPT-5.6-Sol baseline (81.01% accuracy) despite the latter using LLM calls 100% of the time.
  • Error Reduction: Compared to the supervised-only baseline (74.32%), the hybrid approach reduced the error rate by 34.5%.

Limitations and Future Work

  • Rare Class Performance: While the system is more accurate overall, rare classes still suffer from lower recall and F1-scores. Errors in rare classes are often "redirected" by the model to more frequent, similar-looking categories.
  • Open-Set Challenges: The current pipeline assumes the correct code is within the supervised model's candidate list. It does not yet explicitly handle "unseen" parent codes that were not present in the training data.
  • Dataset Specificity: The results are validated on a specific Russian clinical dataset; the authors note that different languages or medical contexts might require retuning the balance between the gate and the reranker.

Future work will likely focus on "rare-class-aware" training strategies, such as hierarchical losses or specialized resampling, to close the gap for less common medical conditions.

Summary updated

Abstract

Automatic assignment of ICD-10 codes from short clinical diagnosis text is a practical medical text classification and normalization task. The task is difficult because diagnosis phrases are short and noisy, the ICD-10 label space is hierarchical, and the class distribution is strongly imbalanced. This paper presents a call-efficient ICD-10 code prediction pipeline that combines transformer embeddings, tree-based supervised candidate generation, candidate-level reranking, and a gated LLM assistance. The evaluation uses a leakage-safe validation protocol: model and threshold selection are based on out-of-fold cross-validation predictions, while final reporting is performed on a locked test set with no shared source-file groups. The supervised conditional parent-code candidate generator reaches 74.32% Top-1 accuracy on the locked test set and 93.05% Top-10 candidate coverage. A direct zero-shot GPT-5.6-Sol baseline reaches 81.01% Top-1 accuracy, but requires an invocation of this large model for every diagnosis. The final deployable system, Pre-LLM fusion, reaches 81.24% OOF-CV accuracy and 83.19% locked-test accuracy while calling the LLM for only 31.97% of cases. This improves the locked-test error rate from 25.68% for the supervised parent-code baseline to 16.81%, corresponding to a 34.5% relative error reduction. The results show that the most effective role of the LLM is controlled reranking on uncertain examples rather than unrestricted coding or full-dataset invocation.
BibTeX
@misc{kopanichuk2026call,
  author = {Ilia Kopanichuk and Daniil Kartashev and Yuri Bolkonsky and Petr Anokhin and Vladimir Makharev and Iaroslav Bespalov and Dmitry V. Dylov},
  title = {{Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation}},
  year = {2026},
  eprint = {2611.21891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21891},
}
Figure