TL;DR
This paper tackles the challenge of making automated clinical coding more transparent and trustworthy by reframing it as an entity linking problem. They propose and evaluate three novel approaches using fine-tuned large language models (LLMs) that not only assign ICD-10 codes but also provide explicit textual evidence, fostering better human-AI collaboration in healthcare.
What Problem Does This Paper Address?
Clinical coding, the process of assigning standardized codes (like ICD-10) to medical records, is crucial for healthcare operations, research, and billing. While automated coding systems exist, they often lack explainability. This means it's hard for medical coders to understand why a code was assigned, making it difficult to trust or verify the system's output. Traditional methods often treat coding as a multi-label classification (MLC) task, which doesn't inherently provide the necessary justification. The paper aims to bridge this gap by making automated coding explainable, enabling medical professionals to critically reason about AI-driven decisions and collaborate more effectively with these tools.
Methods
The core innovation is framing clinical coding as an entity linking problem. Instead of just predicting a code, the system identifies mentions of medical concepts within a report and links them to their corresponding ICD-10 codes, explicitly highlighting the textual evidence.
The authors explore three main approaches, all leveraging the BioMistral 7B large language model:
- ICL-BIOMISTRAL (In-Context Learning): This approach uses a pre-trained BioMistral model and provides it with a prompt that includes the task description, a medical report excerpt, and a mention. Crucially, it also includes a few (10) random examples to guide the model. To ensure valid ICD-10 code outputs, they employ constrained greedy decoding, using a prefix tree to restrict generated tokens to valid ICD-10 code representations.
- SFT-BIOMISTRAL (Supervised Fine-Tuning): Similar to ICL-BIOMISTRAL in its input and output, this model is fine-tuned on a causal language modeling objective. This means the model learns to predict the next token based on the input and previously generated tokens, optimizing for the probability of generating the correct ICD-10 code title. Decoding also uses constrained greedy decoding.
- INSGENEL-BIOMISTRAL: Inspired by the INSGENEL model, this approach aims to output multiple mention-entity pairs for a medical report in a single pass. This mimics how human coders work by considering the document's global context for more coherent predictions. It's fine-tuned using supervised instruction-response pairs and also employs constrained decoding to ensure valid ICD-10 code generation.
Datasets
The study utilizes three publicly available datasets for training and evaluation:
- CodiEsp: Contains Spanish medical reports annotated with ICD-10 codes and textual evidence. It was used for the CodiEsp shared task, which includes automated ICD coding and end-to-end clinical entity linking.
- DisTEMIST: Includes medical reports in Spanish and English, annotated with SNOMED CT disease codes and textual evidence. The authors mapped SNOMED CT codes to ICD-10 using UMLS.
- MDACE: Consists of English medical reports from the MIMIC-III collection, manually annotated with ICD-10 codes and textual evidence spans.
Key Insights and Ideas
- Entity Linking for Explainability: Reframing ICD coding as entity linking is a powerful way to achieve explainability. By linking mentions to codes and providing evidence, the system directly addresses the need for transparency.
- Fine-tuning Outperforms Prompting: Supervised fine-tuning (SFT) and the INSGENEL-inspired approach (INSGENEL-BIOMISTRAL) significantly outperform the in-context learning (ICL) approach, highlighting the benefit of adapting the LLM to the specific task.
- Hierarchical ICD Codes: The ability to predict codes at different hierarchical levels (chapter, subchapter, partial code) is valuable for practical scenarios, offering partial correctness even when the full code isn't identified.
- Few-Shot Learning Potential: The models show promise in handling few-shot scenarios, meaning they can still predict codes that appear infrequently in the training data, which is common in real-world medical coding.
Key Results
- SFT-BIOMISTRAL and INSGENEL-BIOMISTRAL demonstrated stable and strong performance across evaluation corpora, significantly outperforming ICL-BIOMISTRAL.
- INSGENEL-BIOMISTRAL showed particular promise for production scenarios due to its ability to process multiple mentions simultaneously, leading to higher accuracy.
- In hierarchical code prediction, SFT-BIOMISTRAL and INSGENEL-BIOMISTRAL achieved high micro-accuracy at the chapter, subchapter, and partial code levels, offering practical guidance.
- For few-shot analysis, SFT-BIOMISTRAL achieved the best performance, while INSGENEL-BIOMISTRAL also showed capability in predicting few-shot codes.
- When compared to existing systems on the CodiEsp test set, the fine-tuned models achieved similar or better performance in entity linking metrics, even when acknowledging differences in data usage (gold mentions vs. shared task settings).
Limitations and Future Work
- Mention Detection: The current models assume pre-detected mentions. Future work should explore integrating mention detection to create a fully end-to-end solution.
- Dataset Limitations: The experiments were limited to three publicly available datasets, which represent a fraction of patient populations and medical conditions. More diverse and annotated clinical data is needed.
- Computational Cost: Large generative models like BioMistral 7B are computationally expensive. Future research could investigate the impact of using smaller LLMs for efficiency.
- Data Access Restrictions: Due to data access restrictions, the MDACE corpus was used after completing a specific research training course.
- Broader Knowledge Integration: Leveraging additional information from clinical knowledge bases like UMLS could further enhance performance.
Summary updated