Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education

Cynthia K. Y. Ho, Shaowei Guan, Prudence Kwan-Lam Mok, C. Lam, Won Ho, Calvin Hoi‐Kwan Mak, Jing Qin, Arkers Kwan Ching Wong, Vivian HuiJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper introduces NeuroBot, a novel AI chatbot powered by Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) designed to enhance patient education in neurosurgery. By integrating domain-specific knowledge and undergoing rigorous validation, NeuroBot aims to provide accurate, accessible, and timely information to patients, improving their understanding and engagement in perioperative care.

What Problem Does This Paper Address?

Perioperative education is critical for neurosurgery patients, yet traditional methods often fall short due to their labor-intensive nature, lack of scalability, and inability to provide personalized, real-time information. Patients frequently experience anxiety and have complex questions that require tailored answers, which current resources struggle to deliver effectively. While AI chatbots show promise in healthcare, their specific application in the high-stakes and complex field of neurosurgery, particularly regarding their accuracy, relevance, and clinical integration, remains underexplored. This paper addresses this gap by developing and validating a specialized LLM-powered chatbot for neurosurgical perioperative education.

Methods

The study employed a mixed-methods approach across three phases:

  1. Internal Validation: Three LLMs (ChatGPT, Assistants API, and Qwen) were evaluated on 306 bilingual neuroendovascular-related questions. Responses were assessed for accuracy, relevance, and completeness using a Likert scale. The best-performing model was selected for further refinement.
  2. External Validation: The chosen model (Assistants API-based NeuroBot) was evaluated by 10 neurosurgical experts on 50 expert-identified questions. The same metrics (accuracy, relevance, completeness) were used, with consistency measured by intraclass correlation coefficient.
  3. Qualitative Study: 18 healthcare providers (nurses and physicians) participated in interviews to gather insights on NeuroBot's usability, perceived benefits, and potential for clinical integration. Thematic analysis was used to identify key themes.

A key technical component is the Retrieval-Augmented Generation (RAG) framework, which combines LLMs with a domain-specific knowledge base to ensure responses are grounded in factual medical information, mitigating the risk of AI "hallucinations."

Datasets

The study utilized a curated knowledge base for NeuroBot, compiled from:

  • Online medical resources
  • Publications from official health organizations
  • Academic journals
  • Contributions from 6 experienced neurosurgery nurses who generated frequently asked questions (FAQs) and detailed answers.

The internal and external validation phases used a dataset of 306 bilingual neuroendovascular-related questions, derived from real-world clinical encounters. This dataset was translated into both English and traditional Chinese.

Key Insights and Ideas

  • RAG is Crucial for Medical Accuracy: The RAG framework is highlighted as essential for grounding LLM responses in factual medical knowledge, significantly reducing the risk of generating incorrect or fabricated information, a critical concern in healthcare.
  • Domain Expertise is Paramount: The study emphasizes the necessity of involving domain experts (neurosurgical nurses and physicians) not only for initial validation but also for qualitative feedback on usability, relevance, and clinical integration.
  • Patient-Centered Design is Key: The development prioritized a low cognitive load design, responsiveness, and clear visual cues to enhance user engagement and comprehension, especially for patients with complex conditions.
  • LLMs as Communication Bridges: Beyond providing information, NeuroBot is seen as a tool that can act as a communication bridge between patients and healthcare professionals, facilitating better dialogue and shared decision-making.
  • Addressing Anxiety and Self-Efficacy: The chatbot's ability to provide quick, reliable information is expected to calm patient nerves, reduce anxiety, and boost self-efficacy in managing their health.

Key Results

  • Internal Validation: The Assistants API model outperformed ChatGPT and Qwen, achieving a mean accuracy score of 5.28/6 (vs. 4.98/6 for ChatGPT and 2.63/6 for Qwen).
  • External Validation: NeuroBot (Assistants API-based) demonstrated significant improvements, with experts rating it highly:
  • Accuracy: 5.70/6
    • Relevance: 5.58/6
    • Completeness: 2.70/3
    • High interrater agreement (ICC values > 0.95) for all metrics.
  • Qualitative Findings: Healthcare providers perceived NeuroBot as a valuable patient resource tool, capable of saving time, enhancing patient psychological health, improving treatment compliance, and boosting self-efficacy. They also highlighted its potential as a communication aid and preferred its simple, user-friendly design.

Limitations and Future Work

Limitations: * Confirmation Bias: Developers participated in internal validation, potentially introducing bias. External validation by independent experts mitigated this. * Implicit Bias: Lack of formal blinding in some phases might have introduced implicit bias. * Social Desirability Bias: Likert-scale ratings may be influenced by participants wanting to provide socially desirable answers. * Domain Scope: The current knowledge base is limited to neuroendovascular diseases and surgeries. * Data Availability: The chatbot's database is not publicly available.

Future Work: * Expand Domain Scope: Broaden the knowledge base to cover all neurosurgical diseases and surgeries. * Prospective Randomized Controlled Trial: Evaluate NeuroBot's clinical effectiveness in a real-world setting, assessing patient anxiety, knowledge, compliance, and outcomes. * Generalizability: Investigate NeuroBot's adaptability to other neurosurgical centers and specialties by updating the RAG knowledge base. * User-Centered Design: Further refine the interface for the general public, ensuring accessibility, readability, and usability for individuals with varying technological proficiency. * Multimodal Functionality: Incorporate visual aids, voice input/output, and emotional responsiveness. * Ethical Considerations: Rigorously examine ethical compliance, including issues of discrimination, privacy, and the potential for overconfidence or misinterpretation of AI advice. * Response Validation AI: Develop an AI to validate chatbot responses before they are delivered to users. * Real-time Updates: Implement a system for medical professionals to easily update the chatbot with the latest information.

Summary updated

Abstract

NeuroBot, leveraging LLMs with the retrieval-augmented generation technique, demonstrates the potential of LLM-based chatbots in perioperative neuroendovascular care, offering scalable and continuous support. By integrating domain-specific knowledge, NeuroBot simplifies communication between professionals and patients while ensuring patients have 24-7 access to reliable, evidence-based information. Further refinement and research will enhance NeuroBot's ability to foster patient-centered communication, optimize clinical outcomes, and advance AI-driven innovations in health care delivery.
BibTeX
@article{ho2025development,
  author = {Cynthia K. Y. Ho and Shaowei Guan and Prudence Kwan-Lam Mok and C. Lam and Won Ho and Calvin Hoi‐Kwan Mak and Jing Qin and Arkers Kwan Ching Wong and Vivian Hui},
  title = {{Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/74299},
}
Figure