Patient Triage and Guidance in Emergency Departments Using Large Language Models: Multimetric Study

Chenxu Wang, Fei Wang, Shuhan Li, Q W Ren, Xiaomei Tan, Yaoyu Fu, Di Liu, Guangwu Qian, Yu Cao, Rong Yin, Kang LiJournal of Medical Internet Research 2025 · DOI

TL;DR

This study investigates the efficacy of GPT-4-based ChatGPT models (GPT-4o and GPT-4-Turbo) for patient triage and outpatient department guidance in emergency departments (EDs). The findings suggest that while both models show promise, GPT-4-Turbo excels in adaptability to prompt engineering for triage accuracy, whereas GPT-4o demonstrates superior emotional responsiveness, crucial for patient interaction.

What Problem Does This Paper Address?

Emergency departments (EDs) worldwide grapple with significant challenges, including overcrowding, prolonged waiting times, and staff shortages. These issues place immense strain on healthcare systems. Efficient patient triage and accurate departmental guidance are critical to alleviate these pressures. In China, the absence of standardized triage systems and the self-service registration process often lead to confusion, with patients sometimes presenting to EDs for non-urgent issues. This study aims to explore how advanced Large Language Models (LLMs) like ChatGPT can assist in these critical ED processes, specifically in patient triage using the Modified Early Warning Score (MEWS) and in guiding patients to the appropriate outpatient departments.

Methods

The research employed a two-phase experimental design.

Phase 1: Patient Triage Evaluation * Models Tested: GPT-4o and GPT-4-Turbo. * Triage Metric: Modified Early Warning Score (MEWS), a physiological scoring system based on vital signs (blood pressure, heart rate, respiratory rate, temperature) and consciousness level (AVPU score). * Dataset: 1854 simulated patient scenarios derived from existing MEWS research data. * Procedure: * Preliminary Experiment (Trial 1): Models were prompted to act as ED nurses and calculate MEWS scores for each patient. Each scenario was run 10 times to assess consistency. * Prompt Engineering: Techniques were applied to optimize the prompts for improved accuracy and consistency. * Post-Engineering Experiment (Trial 2): The same dataset and procedures were repeated with the refined prompts. * Analysis: Accuracy and consistency were assessed using Wilcoxon rank sum tests, Kendall correlation coefficients, and logistic regression.

Phase 2: Outpatient Department Selection Guidance * Model Tested: GPT-4o (chosen for its reported superior responsiveness and emotional interaction). * Dataset: 264 simulated patient scenarios from the Chinese Medical Case Repository, representing common symptoms and conditions. * Procedure: Each scenario was presented to GPT-4o thrice, independently, to assess consistency. The model was tasked with recommending a specific outpatient department from a predefined list of 33 departments. * Analysis: Accuracy was evaluated by 2-3 physicians. Consistency was assessed using the McNemar test. Logistic regression explored factors influencing guidance correctness.

Datasets

  • Patient Triage: Simulated patient scenarios (n=1854) derived from existing MEWS research data, specifically from Vrije Universiteit University Medical Center.
  • Department Selection Guidance: 264 simulated patient scenarios sourced from the Chinese Medical Case Repository. This repository contains patient case reports and treatments. The authors modified these reports to ensure they reflected typical registration descriptions.

Key Insights and Ideas

  • Prompt Engineering is Crucial: The study strongly highlights the significant impact of prompt engineering on LLM performance in clinical settings. Without it, accuracy and consistency can be suboptimal.
  • GPT-4-Turbo's Adaptability: GPT-4-Turbo demonstrated remarkable adaptability to prompt engineering, achieving perfect accuracy in triage after optimization. This suggests it might be more amenable to fine-tuning for specific tasks.
  • GPT-4o's Emotional Intelligence: GPT-4o showed superior responsiveness and emotional interaction, offering more empathetic and detailed explanations, which are vital for patient-facing applications. This "human-like" quality is highly valued by clinicians.
  • Triage vs. Guidance Nuances: While both models showed promise, their strengths differed. GPT-4-Turbo was better for the objective task of MEWS scoring, while GPT-4o's strengths lay in the more subjective and interactive task of patient guidance.
  • Departmental Accuracy Varies: In department selection, accuracy was not uniform across all departments. Internal medicine and ophthalmology showed high accuracy, while general surgery and dermatology lagged, possibly due to the complexity of symptoms or reliance on visual cues.
  • Limitations of Text-Only Input: The study identified that the inability of LLMs to process visual information (like rashes or physical examinations) is a significant limitation for certain diagnostic tasks, particularly in dermatology.

Key Results

Patient Triage (MEWS Scoring): * Before Prompt Engineering (Trial 1): Both models showed suboptimal performance. GPT-4o had an overall accuracy of 63.41%, and GPT-4-Turbo had 63.93%. * After Prompt Engineering (Trial 2): * GPT-4-Turbo: Achieved an impressive 100% accuracy across all runs. * GPT-4o: Achieved an overall accuracy of 96.20%, with 7 out of 10 runs reaching 100% accuracy. * Consistency: GPT-4-Turbo showed a dramatic improvement in consistency after prompt engineering, reaching a mean Kendall correlation coefficient of 1.00. GPT-4o's consistency also improved but was not as pronounced.

Outpatient Department Selection Guidance (GPT-4o): * Overall Accuracy: 92.63% (95% CI 90.34%-94.93%). * Internal Medicine: Highest accuracy at 93.51% (95% CI 90.85%-96.17%). * General Surgery: Lowest accuracy at 91.46% (95% CI 86.50%-96.43%). * Other Specialized Departments: 93.18% (95% CI 89.68%-96.68%). * Specific Department Examples: Ophthalmology achieved 100% accuracy, while Dermatology had the lowest at 79.57%.

Limitations and Future Work

Limitations: * Chinese Patient Scenarios: The study exclusively used scenarios involving Chinese patients and departments based on Chinese regulations, limiting generalizability. * Limited Department Scope: Only 33 common departments were included, excluding some specialized areas. * Structured Data: Patient scenarios were sourced from a structured medical database, which may not fully represent the variability and potential incompleteness of real-world patient descriptions. * Time-Cost Balance: The practical application of LLMs in time-pressured ED environments needs further verification. * Text-Only Input: The inability to process visual information is a significant constraint for certain medical conditions.

Future Work: * Cross-System and Cross-Cultural Validation: Evaluate ChatGPT's performance in other healthcare systems and with diverse patient populations. * Real-World Patient Descriptions: Incorporate more varied and less structured patient descriptions to better mimic real-world scenarios. * User Satisfaction and Emotional Impact: Empirically evaluate the impact of GPT-4o's emotional responses on user satisfaction. * Prompt Manipulation and Parameter Tuning: Further explore how prompt engineering and parameter tuning can optimize LLM performance. * Integration with Visual Data: Investigate methods for integrating visual data processing into LLM-based clinical tools. * Real-World Implementation: Conduct studies on the actual implementation and impact of these AI tools in clinical settings.

Summary updated

Abstract

ChatGPT demonstrated promising capability for supporting patient triage and outpatient guidance in EDs. GPT-4-Turbo showed greater adaptability to prompt engineering, whereas GPT-4o exhibited superior responsiveness and emotional interaction, which are essential for patient-facing tasks. Future studies should explore real-world implementation and address the identified limitations to enhance ChatGPT's clinical integration.
BibTeX
@article{wang2025patient,
  author = {Chenxu Wang and Fei Wang and Shuhan Li and Q W Ren and Xiaomei Tan and Yaoyu Fu and Di Liu and Guangwu Qian and Yu Cao and Rong Yin and Kang Li},
  title = {{Patient Triage and Guidance in Emergency Departments Using Large Language Models: Multimetric Study}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/71613},
}
Figure