Do Large Language Models understand Medical Codes?

Simon A. Lee, Timothy LindseyarXiv 2024 · OpenReview

TL;DR

This paper investigates whether Large Language Models (LLMs) can accurately interpret and generate descriptions for medical codes, a critical but non-natural language component of healthcare. The study finds that current LLMs struggle significantly with this task, highlighting a gap in their ability to handle structured, coded data, which has implications for automated healthcare processes.

What Problem Does This Paper Address?

Healthcare relies heavily on standardized coded languages like the International Classification of Diseases (ICD-10) for billing, diagnosis, and research. These codes are structured, regularly updated, and deviate from natural language, posing a challenge for LLMs, which are primarily trained on text. The concern is that LLMs might "hallucinate" or generate inaccurate interpretations of these codes, leading to critical errors in patient care, automated billing, and clinical decision support. This paper aims to evaluate how well LLMs understand the underlying structure and ontologies of medical codes and their ability to generate accurate labels from them.

Methods

The researchers designed a series of experiments to assess LLMs' capabilities with medical codes. Their approach involved:

  • Predicting Medical Chapters: Evaluating if LLMs could identify the broader categories (e.g., "Infectious and Parasitic Diseases") associated with specific ICD codes.
  • Predicting Medical Codes: This was broken down into three levels of difficulty:
  • Ordered List: LLMs were given a list of codes in a specific order to predict their corresponding medical conditions. This tests their ability to recognize codes and understand their numerical structure.
  • Randomly Ordered List: Codes were presented in a random order to simulate real-world scenarios where context from adjacent codes might be absent. This tests a more comprehensive understanding.
  • Adversarial Attack: The researchers introduced "fake" ICD codes that mimicked real ones to see if LLMs could distinguish between genuine and fabricated codes, and if they would avoid generating labels for the fake ones.

The study utilized both commercial and open-source LLMs, with a focus on evaluating their performance without internet access during inference to isolate their learned knowledge.

Datasets

The study primarily used publicly available data for medical codes, including:

  • International Classification of Diseases (ICD-9/10): The core dataset for medical diagnoses.
  • Current Procedural Terminology (CPT): Codes for medical procedures.
  • Medications (National Drug Code - NDC): Codes for pharmaceutical drugs.
  • Laboratory Values (LOINC): Codes for laboratory tests.

The paper also references the MIMIC-IV database as a source for demonstrating the frequency and abundance of ICD codes, illustrating the disparity between common and rare codes. No new benchmark datasets were introduced; instead, the focus was on evaluating LLMs' performance on existing coded data.

Key Insights and Ideas

  • LLMs struggle with coded language: The most significant finding is that LLMs, even advanced ones, generally perform poorly when asked to generate accurate labels solely from medical codes. This suggests a fundamental limitation in their ability to process and interpret structured, non-natural language data.
  • Parameter size matters (potentially): GPT-4, being the largest model tested, showed superior performance compared to others. This hints that scaling up model size might be a key factor in improving their ability to handle coded languages, though it's not a complete solution.
  • Awareness of limitations: Many LLMs, like Gemini and Claude, demonstrated an awareness of their limitations by issuing warning messages, advising users to exercise caution. This is a positive sign of their internal confidence calibration.
  • Rarity is a challenge: LLMs are better at identifying common medical codes but struggle significantly with rarer ones. This indicates they learn frequently occurring patterns but lack deeper understanding of less common but potentially critical codes.

Key Results

  • Low Accuracy: Across various experiments, LLMs showed low accuracy in generating correct medical chapter names and predicting medical code names. For instance, in the "Predicting Medical Chapters" task, performance dropped significantly for Procedure and Lab Chapters compared to ICD 9/10 Chapters.
  • Adversarial Attack Failure: LLMs largely failed to distinguish between real and fake ICD codes. While GPT-4 performed slightly better, it still exhibited shortcomings, often erroneously generating labels for fake codes.
  • BERT Score Disparity: A significant performance disparity was observed between ordered and unordered code lists, indicating that LLMs struggle more when codes lack contextual ordering.
  • Model Performance Variance: GPT-4 emerged as the best-performing model, followed by GPT-3.5 and Claude 3. Open-source models like LLaMA-2 and Mistral generally performed worse.

Limitations and Future Work

Limitations:

  • No Internet Access: The study intentionally restricted LLMs from accessing the internet during inference. While this isolates learned knowledge, it might not reflect real-world LLM performance where internet access could aid in retrieving up-to-date information.
  • Reproducibility: The authors acknowledge the inherent variability in LLM outputs due to stochastic decoding methods and periodic model updates, making exact reproduction of results challenging.
  • Focus on Codes Alone: The study focused solely on generating labels from codes, not incorporating broader clinical context, which might be how LLMs are more effectively used in practice.

Future Work:

  • Knowledge Graphs and RAG: Integrating LLMs with knowledge graphs and Retrieval Augmented Generation (RAG) could improve reasoning and information retrieval for coded data.
  • Chain of Thought Reasoning: Employing techniques like Chain of Thought (CoT) reasoning could help LLMs articulate intermediate steps and rationale, leading to better understanding of complex coded structures.
  • Synthetic Data and Feature Engineering: Generating synthetic data or feature-engineered text could fine-tune LLMs to better learn ontologies and differentiate real from fake codes.
  • Scaling: Further research into scaling model parameters could enhance LLMs' capabilities in learning coded languages.

Summary updated

Abstract

Large Language Models (LLMs) have become a pivotal research area, potentially making beneficial contributions in fields like healthcare where they can streamline automated billing and decision support. However, the frequent use of specialized coded languages like ICD-10, which are regularly updated and deviate from natural language formats, presents potential challenges for LLMs in creating accurate and meaningful latent representations. This raises concerns among healthcare professionals about potential inaccuracies or ``hallucinations" that could result in the direct impact of a patient. Therefore, this study evaluates whether large language models (LLMs) are aware of medical code ontologies and can accurately generate names from these codes. We assess the capabilities and limitations of both general and biomedical-specific generative models, such as GPT, LLaMA-2, and Meditron, focusing on their proficiency with domain-specific terminologies. While the results indicate that LLMs struggle with coded language, we offer insights on how to adapt these models to reason more effectively.
BibTeX
@misc{lee2024do,
  author = {Simon A. Lee and Timothy Lindsey},
  title = {{Do Large Language Models understand Medical Codes?}},
  year = {2024},
  eprint = {2403.10822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10822},
}
Figure