Abstract
This study evaluates the reliability of ChatGPT-3.5 and 4.0 in performing outpatient triage—guiding patients to the correct medical department based on their symptoms. While ChatGPT-4.0 is significantly more consistent than its predecessor, the two versions rarely agree with one another, highlighting a critical "version drift" that poses challenges for integrating Large Language Models (LLMs) into standardized clinical workflows.
Introduction
Outpatient departments serve as the primary entry point for healthcare systems, yet department guidance (triage) remains a significant operational bottleneck. While previous research has focused on high-stakes Emergency Department environments, this study explores the "front door" of the hospital. The central challenge for deploying AI in this context is consistency: ensuring that identical symptom descriptions yield stable recommendations and that model upgrades do not disrupt established clinical pathways.
Methods
The researchers employed a comparative study design focusing on "within-version" and "between-version" consistency.
- Datasets: The study used 52 symptom descriptions written in Chinese, sourced from social media (Xiaohongshu) and online medical platforms (Dingxiang Doctor). These reflect natural, "verbalized" patient language rather than sanitized medical text.
- Prompting Strategy: To eliminate memory bias, every query was treated as a fresh, independent chat session. Each symptom was inputted three separate times into both ChatGPT-3.5 and ChatGPT-4.0.
- Grading: Within-version tests tracked identical department recommendations across trials. Between-version tests used a 0–3 point scale based on how many of the top three recommended departments matched between the two versions. Statistical significance was validated using Chi-square tests ($\alpha=0.05$).
Key Results
Within-Version Consistency
ChatGPT-4.0 demonstrated superior internal stability compared to ChatGPT-3.5. Analysis showed that ChatGPT-4.0 was "completely inconsistent" in only 17.3% of its response sets, whereas ChatGPT-3.5 failed to provide consistent answers in 38.5% of cases ($p=0.03$).
Between-Version Disconnect
Despite the improved stability of version 4.0, there was a notable lack of agreement between the two models. The mean consistency score between versions was only 1.43 out of 3. Only 50% of the top recommendations matched perfectly between 3.5 and 4.0, suggesting that model updates fundamentally alter the underlying triage logic.
Discussion and Insights
- The Stability-Speed Tradeoff: While ChatGPT-4.0 is more stable, it is approximately six times slower than GPT-3.5. In high-pressure triage environments, this latency could hinder operational efficiency.
- Version Drift: The low between-version consistency indicates that LLM updates are not merely linear improvements in intelligence but shifts in reasoning. For healthcare providers, an "upgrade" could necessitate a complete re-validation of triage pathways.
- Completeness vs. Consistency: ChatGPT-3.5 was more likely to provide "complete" responses (including urgency levels and probability scores) than GPT-4.0 ($p=0.02$). However, GPT-4.0’s responses were more reliable in their core recommendations.
- Input Quality: The study observed that poorly described symptoms directly led to inconsistent AI outcomes, emphasizing that the integrity of AI triage depends heavily on the quality of patient input.
Conclusion
While ChatGPT-4.0 shows promise for more reliable outpatient triage, the significant version drift and the speed-stability tradeoff remain barriers to clinical adoption. Future integration must focus on tailored prompt engineering and human-factors principles to ensure AI-assisted triage reduces the cognitive load on hospital staff without compromising safety.
Summary updated