TL;DR
This paper introduces Woollie, an open-source, oncology-specific large language model (LLM) trained on clinical radiology reports. Woollie demonstrates high accuracy in predicting cancer progression, outperforming general-purpose LLMs like ChatGPT and even specialized medical LLMs on this critical task, with potential for real-world clinical applications.
What Problem Does This Paper Address?
The adoption of LLMs in healthcare, particularly in oncology, faces several hurdles. These include privacy concerns with closed-source models, performance limitations of general LLMs in specialized medical domains, and the challenge of adapting open-source LLMs to complex, unstructured clinical data like radiology reports. Oncology relies heavily on understanding disease progression, which is often captured in these reports, making accurate interpretation crucial for treatment decisions. Existing medical LLMs often perform well on general medical benchmarks but may not translate to real-world clinical data, and a lack of cross-institutional validation limits their trustworthiness.
Methods
The authors developed Woollie through a "stacked alignment" strategy, building upon Meta's open-source Llama models. This process involved:
- Pre-training: Leveraging large, general-domain datasets (e.g., Common Crawl) to build foundational language understanding in Llama models of various sizes (7B to 65B parameters).
- Foundation Alignment: Supervised fine-tuning of Llama models using diverse datasets (Alpaca, COT, Instruction Wild, OpenAssistant) to enhance reasoning, logic, and conversational abilities, creating "Woollie Foundation" models.
- Medical Alignment: Further supervised fine-tuning with general medical datasets (PubMedQA, MedQuAD, USMLE, MedMCQA) to create "Woollie Medicine" and "Woollie" models, improving medical domain knowledge.
- Oncology Fine-tune: The crucial step involved fine-tuning with a proprietary dataset of 38,719 radiology impression notes from Memorial Sloan Kettering Cancer Center (MSK) across five cancer types. This specialized training resulted in "Woollie MSK" models.
This stacked approach aims to mitigate "catastrophic forgetting" by progressively building domain-specific knowledge while retaining general capabilities.
Datasets
- Training & Alignment:
- Non-medical: Alpaca, Chain of Thoughts (COT), Instruction Wild, OpenAssistant (OASST1 subset), Google's Chain of Thought (COT), and others.
- Medical: PubMedQA, MedQuAD, USMLE, MedMCQA.
- Oncology (Proprietary): 38,719 radiology impression notes from 3402 patients across lung, breast, prostate, pancreatic, and colorectal cancers from Memorial Sloan Kettering Cancer Center (MSK).
- External Validation: 600 radiology impression notes from 600 patients across lung, breast, and prostate cancers from the University of California, San Francisco (UCSF).
Key Insights and Ideas
- Stacked Alignment is Key: The "stacked alignment" strategy is crucial for building specialized LLMs without losing general capabilities, effectively preventing catastrophic forgetting.
- Domain-Specific Data Matters: Fine-tuning on real-world oncology radiology reports significantly boosts performance on cancer progression prediction, even outperforming larger general medical LLMs.
- Cross-Institutional Validation is Essential: Woollie's ability to generalize to an independent UCSF dataset, despite differences in data format and patient demographics, highlights its robustness and potential for wider clinical adoption.
- Smaller Models Can Be Powerful: The 7B Woollie MSK model shows competitive performance, especially in precision, making it a more resource-efficient option for clinical deployment compared to larger models.
- Beyond Binary Classification: Woollie can discern nuanced categories of cancer progression, not just a simple "progressing" vs. "not progressing" label, which is vital for clinical decision-making.
Key Results
- Cancer Progression Prediction (MSK Data):
- Woollie MSK 33B achieved an overall AUROC of 0.90 and accuracy of 0.90.
- Woollie MSK 7B achieved an overall AUROC of 0.86 and accuracy of 0.86.
- Notably, Woollie MSK achieved an AUROC of 0.98 for pancreatic cancer progression.
- Cancer Progression Prediction (UCSF Data):
- Woollie MSK 7B achieved an AUROC of 0.89 and accuracy of 0.80.
- Woollie MSK 33B achieved an AUROC of 0.88 and accuracy of 0.78.
- For lung cancer on UCSF data, Woollie MSK achieved an AUROC of 0.95.
- Comparison to Baselines: Woollie models consistently outperformed baseline Llama models and even ChatGPT on medical benchmarks. For instance, Woollie 65B achieved an accuracy of 0.81 on PubMedQA, surpassing ChatGPT's 0.716.
- Precision: Woollie MSK 33B demonstrated high precision (0.99) in detecting lung cancer progression on the UCSF dataset.
Limitations and Future Work
- Data Specificity: While Woollie excels on radiology impressions, its performance might vary on other types of clinical data. The authors acknowledge that the model's understanding of "hedging language" in radiology reports, which reflects inherent uncertainty, requires further systematic study.
- Multimodality: The current model relies solely on text. Future work could explore multimodal LLMs that integrate imaging data alongside text for more comprehensive analysis.
- Resource Constraints: While the 7B model is more resource-efficient, larger models still offer higher performance, presenting a trade-off for clinical deployment.
- Generalizability Nuances: While cross-institutional validation was successful, performance variations across different cancer types were observed, likely due to reporting style differences. Further research with larger, more balanced cohorts is needed to fully understand performance across demographic subgroups.
- Explainability: While LIME and topic modeling provide insights, further research into LLM explainability in clinical contexts is warranted.
Summary updated