PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models

Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang

TL;DR

This paper introduces PediaBench, the first comprehensive Chinese pediatric dataset designed to rigorously evaluate Large Language Models (LLMs) on their question-answering capabilities in pediatrics. By incorporating diverse question types and a nuanced scoring system, PediaBench aims to reveal the strengths and weaknesses of current LLMs in this specialized medical domain, highlighting areas for future improvement.

What Problem Does This Paper Address?

The rapid advancement of LLMs has spurred interest in their application to the medical field. However, existing medical QA benchmarks often lack depth in specific sub-specialties like pediatrics, or they are limited to objective questions, failing to assess LLMs' generative and reasoning abilities. This gap makes it difficult to comprehensively evaluate LLMs' readiness for real-world pediatric medical applications. PediaBench aims to fill this void by providing a robust, Chinese-language benchmark specifically for pediatric QA.

Methods

The authors constructed PediaBench by collecting questions from various authoritative sources, including the Chinese National Medical Licensing Examination, university medical exams, and pediatric clinical guidelines. The dataset features five question types: True/False (ToF), Multiple Choice (MC), Pairing (PA), Essay/Short Answer (ES), and Case Analysis (CA). To provide a more accurate evaluation, they developed an integrated scoring criterion that considers question difficulty for objective questions and utilizes a Language-Model-as-an-Examiner (LME) framework, specifically employing GPT-4o, for scoring subjective questions. This approach aims to assess instruction following, knowledge understanding, and clinical case analysis skills.

Datasets

The primary contribution of this paper is the introduction of PediaBench, a new benchmark dataset for pediatric question answering in Chinese. PediaBench comprises 4,117 objective questions and 1,632 subjective questions, spanning 12 pediatric disease groups. The dataset was constructed from:

  • Chinese National Medical Licensing Examination (CNMLE)
  • Final exams from medical universities
  • Pediatric Disease Diagnosis and Treatment Standards
  • Pediatric Clinical Guidelines

The paper also references existing medical QA benchmarks such as PubMedQA, MedQA, MedMCQA, CMExam, CMB, MedBench, CBLUE, PromptCBLUE, DDXPlus, and Huatuo-26M, highlighting their limitations in pediatric coverage and subjective question assessment.

Key Insights and Ideas

  • PediaBench is a challenging benchmark: The evaluation results show that even advanced LLMs struggle with PediaBench, indicating its effectiveness in distinguishing model capabilities in pediatric QA.
  • Subjective questions are crucial but difficult: While objective questions test knowledge recall, subjective questions (ES and CA) are vital for assessing reasoning and generation, but LLMs often perform poorly on them, especially in complex case analyses.
  • Pairing questions are highly discriminative: The introduction of pairing questions, which require precise matching, proved to be a challenging task for most LLMs, highlighting their limitations in nuanced understanding.
  • Chinese language proficiency matters: Some general-purpose LLMs, despite having many parameters, underperformed, potentially due to insufficient exposure to Chinese medical corpora during training.
  • Medical LLMs don't always outperform general ones: While specialized medical LLMs are being developed, some general-purpose models, particularly larger ones, showed competitive or superior performance on PediaBench, suggesting that general LLM advancements can be leveraged for medical tasks.

Key Results

The paper evaluated 20 LLMs on PediaBench. Key findings include:

  • Only a few commercial LLMs and Qwen-72B achieved passing overall scores (≥ 60). Qwen-MAX achieved the highest total score of 75.74.
  • Medical LLMs generally received lower scores, indicating the difficulty of the PediaBench dataset and the gap in their instruction-following, reasoning, and generation capacities for pediatric scenarios.
  • Smaller LLMs like Qwen1.5-14B outperformed larger models like GPT3.5-turbo and Mixtral-8x22B in some instances, possibly due to better Chinese language handling.
  • Most LLMs performed better on True/False and Essay/Short Answer questions compared to Pairing and Case Analysis questions.
  • LLaMA and Mixtral models sometimes produced mixed Chinese-English outputs, despite prompt restrictions.
  • LLMs often struggled with factual accuracy, particularly in identifying specific drugs for treatment, sometimes confusing similar-sounding medications.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Scope: PediaBench currently focuses solely on pediatrics and does not encompass all pediatric diseases or their treatments.
  • Scoring Bias: While GPT-4o scoring correlates well with human scoring, there's a potential for scoring style bias.
  • Future Work: The authors plan to expand PediaBench to cover more medical departments. They also suggest exploring more rigorous scoring strategies, such as LLM ensembling, and investigating techniques like Retrieval Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to address LLM limitations in knowledge injection, entity recognition, and reasoning.

Summary updated

Abstract

The emergence of Large Language Models (LLMs) in the medical domain has stressed a compelling need for standard datasets to evaluate their question-answering (QA) performance. Although there have been several benchmark datasets for medical QA, they either cover common knowledge across different departments or are specific to another department rather than pediatrics. Moreover, some of them are limited to objective questions and do not measure the generation capacity of LLMs. Therefore, they cannot comprehensively assess the QA ability of LLMs in pediatrics. To fill this gap, we construct PediaBench, the first Chinese pediatric dataset for LLM evaluation. Specifically, it contains 4,565 objective questions and 1,632 subjective questions spanning 12 pediatric disease groups. It adopts an integrated scoring criterion based on different difficulty levels to thoroughly assess the proficiency of an LLM in instruction following, knowledge understanding, clinical case analysis, etc. Finally, we validate the effectiveness of PediaBench with extensive experiments on 20 open-source and commercial LLMs. Through an in-depth analysis of experimental results, we offer insights into the ability of LLMs to answer pediatric questions in the Chinese context, highlighting their limitations for further improvements. Our code and data are published at https://github.com/ACMISLab/PediaBench.
BibTeX
@article{zhang2024pediabench,
  author = {Qian Zhang and Panfeng Chen and Jiali Li and Linkun Feng and Shuyu Liu and Heng Zhao and Mei Chen and Hui Li and Yanhao Wang},
  title = {{PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models}},
  journal = {Frontiers of Computer Science},
  year = {2024},
  doi = {10.1007/s11704-025-41345-w},
}
Figure