Can Large Language Models Function as Qualified Pediatricians? A Systematic Evaluation in Real-World Clinical Contexts

Siyu Zhu, Mouxiao Bian, Yue Xie, Yongyu Tang, Zhikang Yu, Tianbin Li, Pengcheng Chen, Bing Han, Jie Xu, Xiaoyan DongarXiv 2025 · DOI · arXiv

Introduction

Pediatric medicine presents unique challenges that distinguish it from adult healthcare, including weight-based pharmacology, rapid developmental changes, and the necessity for family-centered communication. While Large Language Models (LLMs) have demonstrated high performance on general medical licensing exams like the USMLE, these assessments often prioritize static factual recall over the dynamic reasoning required in clinical practice. This study introduces PEDIASBench, a comprehensive benchmark designed to evaluate LLMs across the multifaceted demands of pediatric medicine.

The PEDIASBench Framework

To move beyond "medical trivia," the researchers developed a three-dimensional evaluation framework that mirrors the requirements of a qualified pediatrician. The framework encompasses foundational knowledge, dynamic clinical reasoning, and medical ethics.

The PEDIASBench evaluation framework and dataset construction process.

Methodology and Datasets

The PEDIASBench dataset was constructed through a rigorous pipeline involving expert curation and validation:

  1. Foundational Knowledge: 3,665 Q&As derived from standardized medical exams across four professional levels (Resident to Senior Physician), covering 19 subspecialties and 211 diseases.
  2. Dynamic Diagnosis and Treatment: 424 clinical cases curated from real-world anonymized pediatric records. These cases are split into initial consultation (T1) and follow-up (T2) stages to test the model's ability to update reasoning as new data emerges.
  3. Medical Safety and Ethics: 352 Q&As focusing on 10 subcategories, including patient privacy, informed consent via guardians, and humanistic care.

Every item underwent review by at least two pediatricians with over five years of experience, with a senior adjudicator (10+ years) resolving discrepancies.

Performance Analysis

The study evaluated 12 prominent models, including GPT-4o, Qwen3-235B, and DeepSeek-R1, using a zero-shot paradigm to assess their intrinsic capabilities.

Comparative accuracy of LLMs across four levels of pediatric medical examinations.

Key Results

  • Knowledge Proficiency: Top-tier models like Qwen3-235B-A22B achieved over 90% accuracy on licensing-level questions, demonstrating a strong grasp of medical facts.
  • The Reasoning Gap: A significant "complexity tax" was observed; performance dropped by approximately 15% when models transitioned from factual recall to complex clinical reasoning. While DeepSeek-R1 led in case reasoning (mean score of 0.58), most models struggled with "dynamic fragility"—the inability to integrate new lab results or symptoms into a longitudinal diagnosis.
  • Safety and Ethics: Qwen2.5-72B proved most reliable in the safety domain with 92.05% accuracy. However, researchers noted a persistent "humanistic gap," where models lacked the empathetic nuance required for pediatric patient-parent interactions.

Discussion and Future Directions

The evaluation reveals that while LLMs are excellent "students" of medical literature, they are not yet ready to be independent "practitioners." Their reasoning is often brittle when faced with evolving patient conditions. Furthermore, the current text-only evaluation misses the multimodal nature of pediatrics, where physical observation and imaging are critical.

Future work should focus on multimodal integration (incorporating imaging and signals) and establishing a "clinical feedback loop" where models iterate based on real-world physician corrections to enhance safety, trust, and humanistic sensitivity.

Summary updated

Abstract

With the rapid rise of large language models (LLMs) in medicine, a key question is whether they can function as competent pediatricians in real-world clinical settings. We developed PEDIASBench, a systematic evaluation framework centered on a knowledge-system framework and tailored to realistic clinical environments. PEDIASBench assesses LLMs across three dimensions: application of basic knowledge, dynamic diagnosis and treatment capability, and pediatric medical safety and medical ethics. We evaluated 12 representative models released over the past two years, including GPT-4o, Qwen3-235B-A22B, and DeepSeek-V3, covering 19 pediatric subspecialties and 211 prototypical diseases. State-of-the-art models performed well on foundational knowledge, with Qwen3-235B-A22B achieving over 90% accuracy on licensing-level questions, but performance declined ~15% as task complexity increased, revealing limitations in complex reasoning. Multiple-choice assessments highlighted weaknesses in integrative reasoning and knowledge recall. In dynamic diagnosis and treatment scenarios, DeepSeek-R1 scored highest in case reasoning (mean 0.58), yet most models struggled to adapt to real-time patient changes. On pediatric medical ethics and safety tasks, Qwen2.5-72B performed best (accuracy 92.05%), though humanistic sensitivity remained limited. These findings indicate that pediatric LLMs are constrained by limited dynamic decision-making and underdeveloped humanistic care. Future development should focus on multimodal integration and a clinical feedback-model iteration loop to enhance safety, interpretability, and human-AI collaboration. While current LLMs cannot independently perform pediatric care, they hold promise for decision support, medical education, and patient communication, laying the groundwork for a safe, trustworthy, and collaborative intelligent pediatric healthcare system.
BibTeX
@misc{zhu2025can,
  author = {Siyu Zhu and Mouxiao Bian and Yue Xie and Yongyu Tang and Zhikang Yu and Tianbin Li and Pengcheng Chen and Bing Han and Jie Xu and Xiaoyan Dong},
  title = {{Can Large Language Models Function as Qualified Pediatricians? A Systematic Evaluation in Real-World Clinical Contexts}},
  year = {2025},
  eprint = {2511.13381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.13381},
}
Figure