Me LLaMA: Foundation Large Language Models for Medical Applications

Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Kuttichi Keloth, Xingyu Zhou, Huan He, Lucila Ohno-Machado, Yonghui Wu, Hua Xu, Jiang Bian

Abstract

Me-LLaMA is a new family of open-source medical large language models (13B and 70B) built by continually pre-training LLaMA2 on a massive 129-billion-token corpus of biomedical literature and clinical notes. By combining formal scientific knowledge with real-world clinical data, Me-LLaMA achieves state-of-the-art performance across diverse medical text analysis tasks and rivals commercial giants like GPT-4 in complex clinical diagnosis.

Introduction

While general-purpose large language models (LLMs) have demonstrated significant potential in healthcare, they frequently lack the specialized, nuanced knowledge required for high-stakes medical applications. Existing open-source medical models typically face three primary challenges: data silos (training on either literature or clinical notes, but rarely both), narrow evaluation (focusing primarily on Question Answering), and catastrophic forgetting (losing general reasoning abilities during domain-specific training).

Me-LLaMA was developed to bridge these gaps, creating a foundation model capable of handling a broad spectrum of tasks, from scientific summarization to bedside diagnostic reasoning.

Methodology

The researchers utilized LLaMA2 (13B and 70B versions) as the backbone and implemented a two-stage training pipeline.

Overview of the Me-LLaMA development pipeline, including continual pre-training, instruction fine-tuning, and multi-task evaluation.

Continual Pre-training

Domain knowledge was injected into the base models using 129 billion tokens. A critical aspect of this stage was the use of a 15:1:4 ratio of biomedical literature, clinical notes, and general domain data. This specific mix was designed to mitigate knowledge forgetting while maximizing medical expertise. The authors noted that a 4:1 ratio of medical to general data represents a "sweet spot" for maintaining general reasoning performance.

Instruction Tuning

To ensure the models were "chat-ready" and capable of following complex medical prompts, they were fine-tuned on a novel dataset of 214,595 high-quality samples. This dataset included medical flashcards, clinical guidelines, and knowledge graph relations. The training employed LoRA (Low-Rank Adaptation) for parameter efficiency, requiring over 100,000 A100 GPU hours for the 70B models.

Evaluation Framework

The study introduced the Medical Instruction Benchmark Evaluation (MIBE), a comprehensive suite covering 12 datasets across six core tasks:

  • Question Answering (QA): PubMedQA, MedQA, MedMCQA, EmrQA.
  • Information Extraction: i2b2 (Named Entity Recognition) and DDI (Relation Extraction).
  • Classification: Hallmarks of Cancer (HoC) and MTSample.
  • Summarization: PubMed and MIMIC-CXR.
  • Natural Language Inference (NLI): BioNLI and MedNLI.

Additionally, the models were evaluated on 70 complex clinical cases from the New England Journal of Medicine (NEJM) Clinicopathologic Conferences (CPCs) to assess real-world clinical utility.

Key Results and Insights

Me-LLaMA demonstrated significant performance gains over existing open-source and commercial models.

Performance comparison of Me-LLaMA against LLaMA2, PMC-LLaMA, and Meditron across various medical benchmarks.

Supervised and Zero-Shot Performance

Me-LLaMA 13B outperformed the previous state-of-the-art (PMC-LLaMA 13B) on 11 out of 12 datasets. In zero-shot settings, Me-LLaMA 70B-chat outperformed ChatGPT on 7 out of 8 datasets and GPT-4 on 5 out of 8 datasets.

Clinical Diagnostic Accuracy

On the challenging NEJM cases, Me-LLaMA’s diagnostic accuracy was comparable to GPT-4. Human evaluations by clinicians indicated that Me-LLaMA showed a slight edge in top-1 and top-5 accuracy over GPT-4, highlighting the value of integrating de-identified clinical notes from sources like MIMIC-III/IV.

Conclusion and Future Work

Me-LLaMA represents a significant step toward open-source parity with closed-source models in specialized vertical fields. However, limitations remain, including the potential for hallucinations and a restricted 4,096-token context window inherited from LLaMA2. Future work will focus on incorporating Reinforcement Learning from Human Feedback (RLHF) to better align responses with medical truth and expanding the context window for longitudinal patient care analysis.

The Me-LLaMA models and evaluation scripts are publicly available, providing a robust resource for the medical AI community.

Summary updated

BibTeX
@misc{xie2024me,
  author = {Qianqian Xie and Qingyu Chen and Aokun Chen and Cheng Peng and Yan Hu and Fongci Lin and Xueqing Peng and Jimin Huang and Jeffrey Zhang and Vipina Kuttichi Keloth and Xingyu Zhou and Huan He and Lucila Ohno-Machado and Yonghui Wu and Hua Xu and Jiang Bian},
  title = {{Me LLaMA: Foundation Large Language Models for Medical Applications}},
  year = {2024},
  eprint = {2402.12749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.12749},
}
Figure