Fine-Tuning In-House Large Language Models to Infer Differential Diagnosis from Radiology Reports

Luoyao Chen, Revant Teotia, Antonio Verdone, Aidan Cardall, Lakshay Tyagi, Yiqiu Shen, Sumit Chopra

TL;DR

This paper introduces a pipeline to fine-tune small, open-source large language models (LLMs) to extract differential diagnoses from unstructured radiology reports. By leveraging knowledge distillation from GPT-4 and QLoRA fine-tuning, the authors train 8B-parameter models that match or exceed GPT-4’s performance while completely eliminating the privacy risks and high costs associated with commercial APIs.

What Problem Does This Paper Address?

Radiology reports contain critical clinical insights, particularly in the "impression" section where radiologists synthesize findings into a differential diagnosis. Automatically extracting these diagnoses is vital for downstream clinical decision support, research, and billing compliance.

However, doing this automatically is challenging because radiology reports are highly unstructured, filled with diverse linguistic styles, and prone to inconsistent formatting. While proprietary models like GPT-4 excel at extracting this information, deploying them in real-world healthcare settings introduces two massive roadblocks:

  1. Data Privacy: Sending Protected Health Information (PHI) to external commercial APIs risks violating strict healthcare regulations (like HIPAA).
  2. Cost: Processing thousands of daily radiology reports through commercial APIs is financially prohibitive at scale.

The authors address this by exploring whether smaller, open-source models can be trained to run locally (in-house) without sacrificing clinical accuracy.

Methods

The authors developed a two-step pipeline to build high-performing, in-house clinical LLMs without requiring thousands of expensive human annotations:

  1. Automated Label Generation (Knowledge Distillation): They used GPT-4-32K to process clinical impressions and label them based on a predefined catalog of 133 pathologies. To ensure label reliability, they ran GPT-4 three times per report and used majority voting.
  2. QLoRA Fine-Tuning: Using the GPT-generated dataset, they fine-tuned several open-source models (including Llama-3-8B and Mistral-7B-v0.3). They utilized Quantized Low-Rank Adaptation (QLoRA) with 4-bit precision to drastically reduce memory usage and computational costs, making training feasible on standard hardware (four A100 GPUs).
Figure 1: Pipeline for fine-tuning in-house LLM

Datasets

The study utilized a dataset of 31,056 radiology reports from the musculoskeletal (MSK) imaging department of a large academic medical center.

  • Scope: The dataset covers multiple modalities (MRI, ultrasound, CT, X-ray) and various anatomical regions (knee, wrist, pelvis, shoulder, spine, etc.).
  • Pathology Catalog: The authors defined a taxonomy of 133 differential diagnoses spanning tendon, ligament, bone, joint, infectious, and neoplastic pathologies.
  • Evaluation Set: For a true "gold standard" evaluation, a separate set of 1,067 impressions was meticulously annotated and refined by human radiologists to evaluate the final models against GPT-4.

Key Insights and Ideas

  • The Power of Specialization: An 8-billion parameter model, when properly fine-tuned on a specific domain, can outperform a closed-source model that is estimated to be over 100 times larger (GPT-4 is rumored to be ~1.8 trillion parameters).
  • Zero-Shot Failure vs. Fine-Tuned Success: Out-of-the-box open-source models are virtually useless for this highly specialized task. For instance, vanilla Llama-3-8B achieved a dismal 3.6% Micro F1 score without fine-tuning. Post-fine-tuning, it skyrocketed to 91.9%.
  • Generative > Discriminative: Fine-tuned generative LLMs significantly outperformed fine-tuned discriminative models like BERT. This suggests that the generative pre-training phase gives LLMs a superior foundational understanding of complex clinical context and negation handling.

Key Results

The fine-tuned in-house models didn't just match GPT-4—they frequently beat it, particularly in Recall (the ability to catch all relevant diagnoses).

  • Top Performer: The fine-tuned Mistral-v0.3 achieved the highest overall Micro F1 score of 92.6%, outperforming GPT-4's score of 90.8%.
  • Modality Consistency: The performance improvements were consistent across different imaging modalities (Computed Radiography [CR], CT, and MRI), as shown in the breakdown of F1 scores.
Figure 2: F1-micro across different modalities
Performance of baseline and fine-tuned models

Limitations and Future Work

Despite the impressive results, the authors noted a few edge cases where the in-house models faltered during error analysis:

  • Anatomical and Symptomatic Overlap: The models occasionally confused highly similar pathologies that share proximity and text descriptions. For example, mispredicting a partial rotator cuff tear as a biceps tear because both occur in the shoulder, share symptoms (pain, weakness), and present similar radiographic descriptions.

Future Directions: The current study was strictly limited to musculoskeletal (MSK) reports. The authors plan to expand and generalize this pipeline to other critical radiology subdomains, such as Neuroradiology, Oncologic Imaging, and Pediatric Radiology.

Summary updated

Abstract

Radiology reports summarize key findings and differential diagnoses derived from medical imaging examinations. The extraction of differential diagnoses is crucial for downstream tasks, including patient management and treatment planning. However, the unstructured nature of these reports, characterized by diverse linguistic styles and inconsistent formatting, presents significant challenges. Although proprietary large language models (LLMs) such as GPT-4 can effectively retrieve clinical information, their use is limited in practice by high costs and concerns over the privacy of protected health information (PHI). This study introduces a pipeline for developing in-house LLMs tailored to identify differential diagnoses from radiology reports. We first utilize GPT-4 to create 31,056 labeled reports, then fine-tune open source LLM using this dataset. Evaluated on a set of 1,067 reports annotated by clinicians, the proposed model achieves an average F1 score of 92.1\%, which is on par with GPT-4 (90.8\%). Through this study, we provide a methodology for constructing in-house LLMs that: match the performance of GPT, reduce dependence on expensive proprietary models, and enhance the privacy and security of PHI.
BibTeX
@misc{chen2024fine,
  author = {Luoyao Chen and Revant Teotia and Antonio Verdone and Aidan Cardall and Lakshay Tyagi and Yiqiu Shen and Sumit Chopra},
  title = {{Fine-Tuning In-House Large Language Models to Infer Differential Diagnosis from Radiology Reports}},
  year = {2024},
  eprint = {2410.09234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.09234},
}
Figure