Abstract
Me-LLaMA is a new family of open-source medical large language models (13B and 70B) built by continually pre-training LLaMA2 on a massive 129-billion-token corpus of biomedical literature and clinical notes. By combining formal scientific knowledge with real-world clinical data, Me-LLaMA achieves state-of-the-art performance across diverse medical text analysis tasks and rivals commercial giants like GPT-4 in complex clinical diagnosis.
Introduction
While general-purpose large language models (LLMs) have demonstrated significant potential in healthcare, they frequently lack the specialized, nuanced knowledge required for high-stakes medical applications. Existing open-source medical models typically face three primary challenges: data silos (training on either literature or clinical notes, but rarely both), narrow evaluation (focusing primarily on Question Answering), and catastrophic forgetting (losing general reasoning abilities during domain-specific training).
Me-LLaMA was developed to bridge these gaps, creating a foundation model capable of handling a broad spectrum of tasks, from scientific summarization to bedside diagnostic reasoning.
Methodology
The researchers utilized LLaMA2 (13B and 70B versions) as the backbone and implemented a two-stage training pipeline.
Continual Pre-training
Domain knowledge was injected into the base models using 129 billion tokens. A critical aspect of this stage was the use of a 15:1:4 ratio of biomedical literature, clinical notes, and general domain data. This specific mix was designed to mitigate knowledge forgetting while maximizing medical expertise. The authors noted that a 4:1 ratio of medical to general data represents a "sweet spot" for maintaining general reasoning performance.
Instruction Tuning
To ensure the models were "chat-ready" and capable of following complex medical prompts, they were fine-tuned on a novel dataset of 214,595 high-quality samples. This dataset included medical flashcards, clinical guidelines, and knowledge graph relations. The training employed LoRA (Low-Rank Adaptation) for parameter efficiency, requiring over 100,000 A100 GPU hours for the 70B models.
Evaluation Framework
The study introduced the Medical Instruction Benchmark Evaluation (MIBE), a comprehensive suite covering 12 datasets across six core tasks:
- Question Answering (QA): PubMedQA, MedQA, MedMCQA, EmrQA.
- Information Extraction: i2b2 (Named Entity Recognition) and DDI (Relation Extraction).
- Classification: Hallmarks of Cancer (HoC) and MTSample.
- Summarization: PubMed and MIMIC-CXR.
- Natural Language Inference (NLI): BioNLI and MedNLI.
Additionally, the models were evaluated on 70 complex clinical cases from the New England Journal of Medicine (NEJM) Clinicopathologic Conferences (CPCs) to assess real-world clinical utility.
Key Results and Insights
Me-LLaMA demonstrated significant performance gains over existing open-source and commercial models.
Supervised and Zero-Shot Performance
Me-LLaMA 13B outperformed the previous state-of-the-art (PMC-LLaMA 13B) on 11 out of 12 datasets. In zero-shot settings, Me-LLaMA 70B-chat outperformed ChatGPT on 7 out of 8 datasets and GPT-4 on 5 out of 8 datasets.
Clinical Diagnostic Accuracy
On the challenging NEJM cases, Me-LLaMA’s diagnostic accuracy was comparable to GPT-4. Human evaluations by clinicians indicated that Me-LLaMA showed a slight edge in top-1 and top-5 accuracy over GPT-4, highlighting the value of integrating de-identified clinical notes from sources like MIMIC-III/IV.
Conclusion and Future Work
Me-LLaMA represents a significant step toward open-source parity with closed-source models in specialized vertical fields. However, limitations remain, including the potential for hallucinations and a restricted 4,096-token context window inherited from LLaMA2. Future work will focus on incorporating Reinforcement Learning from Human Feedback (RLHF) to better align responses with medical truth and expanding the context window for longitudinal patient care analysis.
The Me-LLaMA models and evaluation scripts are publicly available, providing a robust resource for the medical AI community.
Summary updated