TL;DR
This paper introduces a pipeline to fine-tune small, open-source large language models (LLMs) to extract differential diagnoses from unstructured radiology reports. By leveraging knowledge distillation from GPT-4 and QLoRA fine-tuning, the authors train 8B-parameter models that match or exceed GPT-4’s performance while completely eliminating the privacy risks and high costs associated with commercial APIs.
What Problem Does This Paper Address?
Radiology reports contain critical clinical insights, particularly in the "impression" section where radiologists synthesize findings into a differential diagnosis. Automatically extracting these diagnoses is vital for downstream clinical decision support, research, and billing compliance.
However, doing this automatically is challenging because radiology reports are highly unstructured, filled with diverse linguistic styles, and prone to inconsistent formatting. While proprietary models like GPT-4 excel at extracting this information, deploying them in real-world healthcare settings introduces two massive roadblocks:
- Data Privacy: Sending Protected Health Information (PHI) to external commercial APIs risks violating strict healthcare regulations (like HIPAA).
- Cost: Processing thousands of daily radiology reports through commercial APIs is financially prohibitive at scale.
The authors address this by exploring whether smaller, open-source models can be trained to run locally (in-house) without sacrificing clinical accuracy.
Methods
The authors developed a two-step pipeline to build high-performing, in-house clinical LLMs without requiring thousands of expensive human annotations:
- Automated Label Generation (Knowledge Distillation): They used GPT-4-32K to process clinical impressions and label them based on a predefined catalog of 133 pathologies. To ensure label reliability, they ran GPT-4 three times per report and used majority voting.
- QLoRA Fine-Tuning: Using the GPT-generated dataset, they fine-tuned several open-source models (including Llama-3-8B and Mistral-7B-v0.3). They utilized Quantized Low-Rank Adaptation (QLoRA) with 4-bit precision to drastically reduce memory usage and computational costs, making training feasible on standard hardware (four A100 GPUs).
Datasets
The study utilized a dataset of 31,056 radiology reports from the musculoskeletal (MSK) imaging department of a large academic medical center.
- Scope: The dataset covers multiple modalities (MRI, ultrasound, CT, X-ray) and various anatomical regions (knee, wrist, pelvis, shoulder, spine, etc.).
- Pathology Catalog: The authors defined a taxonomy of 133 differential diagnoses spanning tendon, ligament, bone, joint, infectious, and neoplastic pathologies.
- Evaluation Set: For a true "gold standard" evaluation, a separate set of 1,067 impressions was meticulously annotated and refined by human radiologists to evaluate the final models against GPT-4.
Key Insights and Ideas
- The Power of Specialization: An 8-billion parameter model, when properly fine-tuned on a specific domain, can outperform a closed-source model that is estimated to be over 100 times larger (GPT-4 is rumored to be ~1.8 trillion parameters).
- Zero-Shot Failure vs. Fine-Tuned Success: Out-of-the-box open-source models are virtually useless for this highly specialized task. For instance, vanilla Llama-3-8B achieved a dismal 3.6% Micro F1 score without fine-tuning. Post-fine-tuning, it skyrocketed to 91.9%.
- Generative > Discriminative: Fine-tuned generative LLMs significantly outperformed fine-tuned discriminative models like BERT. This suggests that the generative pre-training phase gives LLMs a superior foundational understanding of complex clinical context and negation handling.
Key Results
The fine-tuned in-house models didn't just match GPT-4—they frequently beat it, particularly in Recall (the ability to catch all relevant diagnoses).
- Top Performer: The fine-tuned Mistral-v0.3 achieved the highest overall Micro F1 score of 92.6%, outperforming GPT-4's score of 90.8%.
- Modality Consistency: The performance improvements were consistent across different imaging modalities (Computed Radiography [CR], CT, and MRI), as shown in the breakdown of F1 scores.
Limitations and Future Work
Despite the impressive results, the authors noted a few edge cases where the in-house models faltered during error analysis:
- Anatomical and Symptomatic Overlap: The models occasionally confused highly similar pathologies that share proximity and text descriptions. For example, mispredicting a partial rotator cuff tear as a biceps tear because both occur in the shoulder, share symptoms (pain, weakness), and present similar radiographic descriptions.
Future Directions: The current study was strictly limited to musculoskeletal (MSK) reports. The authors plan to expand and generalize this pipeline to other critical radiology subdomains, such as Neuroradiology, Oncologic Imaging, and Pediatric Radiology.
Summary updated