Abstract
As parents increasingly turn to artificial intelligence for medical guidance, the reliability of Large Language Models (LLMs) in clinical contexts becomes paramount. This study evaluates four prominent LLMs—ChatGPT-3.5, ChatGPT-4.0, YouChat, and Perplexity—on their ability to manage pediatric fever. Through a "two-layer" evaluation involving both pediatricians and patient relatives, the research reveals that internet-augmented models do not necessarily outperform static models. Furthermore, a significant gap exists between expert medical accuracy and the practical utility perceived by laypeople, who often struggle to distinguish between high- and low-quality medical advice.
Introduction
Pediatric fever is a primary driver of parental anxiety and healthcare consultations. While LLMs have demonstrated the ability to pass standardized medical examinations, their utility in real-world patient management remains under-explored. This study addresses the critical gap between theoretical medical knowledge and actionable clinical communication. Specifically, it investigates whether real-time internet search capabilities improve the quality of advice and how the "layperson" experience differs from professional medical evaluation.
Methodology
The researchers implemented a robust evaluation framework consisting of two distinct layers:
- Question Sourcing: 30 core questions were derived from the Chinese Journal of Applied Clinical Pediatrics, covering topics ranging from basic temperature measurement to complex medication management for conditions like Kawasaki disease.
- Model Selection: The study compared "static" models (ChatGPT-3.5 and ChatGPT-4.0) against internet-augmented models (YouChat and Perplexity).
- The Professional Layer: 20 experienced pediatricians evaluated model responses across four dimensions: Accuracy, Correctness, Completeness, and Practicality using a 10-point Likert scale.
- The Relative Layer: 120 relatives of pediatric patients assessed the same responses based on Match (relevance), Clarity, Doubt-addressing, and Comfort.
- Statistical Analysis: Tukey post-hoc tests were utilized to identify statistically significant differences between model performances and evaluation dimensions.
Key Results and Insights
The Internet Search Paradox
Contrary to the hypothesis that real-time data would improve performance, internet-enabled models (YouChat and Perplexity) performed significantly worse than the static ChatGPT models according to medical experts. The researchers suggest that for established medical guidelines, real-time web searches may introduce "noise" from unreliable sources, whereas the high-quality pre-training data of GPT-4.0 provides a more stable and accurate foundation for clinical advice.
The Expert-Layperson Divergence
A primary finding of the study was the stark contrast in perception between doctors and parents. While pediatricians identified clear performance leaders, parents rated almost all models similarly. This suggests a "medical literacy gap" where non-professionals may lack the technical knowledge required to identify subtle inaccuracies or omissions, potentially leading to a false sense of security when using AI tools.
Accuracy vs. Practicality
Across all models, "Accuracy" consistently received the highest scores, while "Practicality"—the ability to provide actionable solutions for daily care—was the lowest-rated dimension. Qualitatively, LLMs tended to provide "balanced" answers that listed various pros and cons without offering a definitive "bottom line." While scientifically rigorous, this lack of clear direction is often unhelpful for parents in high-stress medical situations.
Conclusion and Future Directions
The study concludes that while LLMs show promise, they are currently better at reciting facts than managing patients. The inability of laypeople to critically evaluate AI-generated medical content poses a risk of misinformation.
Future development should focus on Retrieval-Augmented Generation (RAG), which forces models to reference specific, authoritative medical databases rather than the open web. Additionally, improving the "actionability" of responses—ensuring they provide clear, understandable conclusions—is essential for making AI a safe and effective tool for patient-facing medical communication.
Summary updated