NurseLLM: The First Specialized Language Model for Nursing

Md Tawkat Islam Khondaker, Julia Harrington, Shady ShehataEMNLP 2025

TL;DR

This paper introduces NurseLLM, the first large language model (LLM) specifically designed for nursing question-answering tasks. By developing a novel, multi-stage data generation pipeline and creating specialized benchmarks, they demonstrate that NurseLLM significantly outperforms existing general and medical LLMs on nursing-specific evaluations, highlighting the critical need for domain-specific AI in healthcare.

What Problem Does This Paper Address?

While LLMs have made significant strides in general and medical AI applications, their potential in the specialized domain of nursing remains largely underexplored. Nursing, with its unique focus on holistic, person-centered care, differs from the biomedical model of medicine. This distinction means that generic or even medical-specialized LLMs may not adequately capture the nuances of nursing practice, decision-making, and knowledge. Furthermore, existing benchmarks for evaluating LLMs in medicine don't fully address the specific requirements of nursing exams like the NCLEX, leaving a gap in rigorous assessment for nursing AI tools.

Methods

The core contribution is the development of NurseLLM, a specialized LLM for nursing question-answering. This involved:

  • Multi-stage Data Generation Pipeline: A novel pipeline was created to build a large-scale, topic-diverse nursing question-answering (QA) dataset. This pipeline starts with creating a comprehensive taxonomy of nursing topics, then uses GPT-40 to generate synthetic NCLEX-style multiple-choice questions (MCQs) with rationales.
  • Chain-of-Thought (CoT) Reasoning: The generated rationales were used to train NurseLLM with CoT reasoning, enhancing its ability to explain its answers.
  • Specialized Benchmarks: Three new benchmarks were developed for rigorous evaluation:
  • NCLEX-Test: A human-labeled gold standard dataset of 1,726 NCLEX questions.
  • GPT40-Test: A synthetic benchmark of 1,830 GPT-40 generated NCLEX MCQs.
  • MultiNurseQA: A subset of the MultiMedQA dataset, specifically filtered for nursing-exclusive questions.
  • Model Training and Merging: NurseLLM was finetuned on the Llama3-Med42-8B base model using QLoRA for efficiency. To mitigate catastrophic forgetting, a model merging technique (DARE) was employed.
  • Exploration of Advanced Techniques: The paper also investigates the impact of reasoning-incorporated training and explores NurseLLM's potential within a multi-agent system (MAS) for nursing QA.

Datasets

  • Training Data: A large-scale synthetic NCLEX-equivalent nursing MCQ dataset comprising 125K samples, generated through the multi-stage pipeline.
  • Evaluation Benchmarks:
    • NCLEX-Test: A human-labeled dataset of 1,726 NCLEX questions.
  • GPT40-Test: A GPT-40 generated dataset of 1,830 NCLEX MCQs.
  • MultiMedQA: A comprehensive suite of medical QA datasets.
  • MultiNurseQA: A subset of MultiMedQA specifically curated for nursing questions.

Key Insights and Ideas

  • Domain Specialization is Crucial: The most significant insight is that a nursing-specific LLM (NurseLLM) significantly outperforms general and even medical-specialized LLMs on nursing tasks. This underscores that the unique knowledge and reasoning patterns in nursing require dedicated models.
  • Synthetic Data Quality Matters: The multi-stage data generation pipeline, which includes generating rationales and ensuring topic diversity, proved effective in creating a high-quality dataset that contributes to NurseLLM's superior performance.
  • Reasoning Enhances Performance (but at a Cost): Incorporating reasoning during training shows promise, but it comes with increased computational time. This suggests a trade-off between accuracy and efficiency that needs careful consideration.
  • Multi-Agent Systems Offer Further Improvement: The exploration of NurseLLM within a multi-agent system demonstrated an improvement in accuracy, suggesting that collaborative AI approaches can further enhance performance in complex nursing scenarios.

Key Results

  • NCLEX-Test Performance: NurseLLM achieved over 76% accuracy, outperforming all other open-source LLMs, including medical-specialized ones like Med42 (72.41%) and OpenBioLLM (71.99%).
  • GPT40-Test Performance: NurseLLM (8B parameters) achieved over 91% accuracy, closely matching GPT-40-mini (94.97%), while other baselines struggled.
  • MultiNurseQA Performance: NurseLLM achieved the second-best performance, only slightly trailing Med42, and showed the smallest performance degradation compared to other models when moving from general medical to nursing-specific questions.
  • Reasoning Impact: A reasoning-incorporated version of NurseLLM scored 72.01% on NCLEX-Test, demonstrating the potential of reasoning, even with a smaller dataset.
  • Multi-Agent System (MAS) Impact: The MAS approach improved accuracy to 79.87% on NCLEX-Test, a 3.62% improvement over NurseLLM.

Limitations and Future Work

  • Data Generation Timeline: The performance of GPT-40 can change over time, meaning the prompting strategies and data generation pipeline might need adjustments.
  • Data Quality: While the automated pipeline is scalable, there's a risk of low-quality data. Human inspection is recommended to ensure robustness and maintain quality.
  • Model Responses: LLMs can still produce inaccurate responses, and NurseLLM should be used with caution in clinical settings.
  • Model Evaluation: The study only evaluated open-source models due to comparability issues with closed-source models and proprietary benchmark concerns.
  • Future Work:
    • Expanding the reasoning-incorporated dataset to develop a specialized nursing reasoning model.
  • Further exploring multi-agent systems for nursing applications.
  • Releasing synthetic test benchmarks and the training dataset.

Summary updated

BibTeX
@inproceedings{khondaker2025nursellm,
  author = {Md Tawkat Islam Khondaker and Julia Harrington and Shady Shehata},
  title = {{NurseLLM: The First Specialized Language Model for Nursing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure