Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies

Luyi Jiang, Jiayuan Chen, Lu Lu, Xinwei Peng, Lihao Liu, Junjun He, Jie XuarXiv 2025 · DOI · arXiv

Introduction

While general-purpose Large Language Models (LLMs) have demonstrated impressive capabilities on standard benchmarks, their transition into the high-stakes clinical environment remains fraught with challenges. In medicine, "knowing" a fact is fundamentally different from "reasoning" through a complex patient case. This study provides a rigorous evaluation of the current state of Chinese medical LLMs, moving beyond macro-level metrics to dissect the specific failure modes that hinder clinical deployment.

Methodology: The MedBench Framework

Researchers evaluated 10 leading Chinese medical LLMs using MedBench, a comprehensive system spanning five critical dimensions: language comprehension, generation, understanding, complex reasoning, and safety/ethics. The evaluation employed two distinct methodologies:

  1. Objective Evaluation: To test for positional bias, multiple-choice questions were subjected to "option shuffling." A model was deemed "robust" only if it consistently selected the correct answer across five different permutations.
  2. Subjective Evaluation: For open-ended responses, a three-expert consensus protocol was used to categorize errors into an 8-type granular taxonomy, including Omissions, Hallucinations, and Causal Reasoning Deficiencies.

The "Omission" Epidemic and Reasoning Gaps

The most significant finding of the study is that the primary failure mode of current models is not fabrication (hallucination), but rather the omission of critical information. Omissions accounted for nearly 40% of all errors across the evaluated tasks.

Overall distribution of error types identified across the top 10 Chinese medical LLMs.

This issue becomes particularly acute in the context of Complex Medical Reasoning, where the omission rate skyrocketed to 96.3%. While models can often recall basic medical facts, they struggle to synthesize these facts into a complete diagnostic or treatment plan, frequently providing coherent-sounding but dangerously incomplete advice.

Breakdown of error distributions by medical task dimension, highlighting the extreme omission rate in complex reasoning tasks.

Performance Disconnects: Knowledge vs. Logic

The data reveals a stark divide between memory and logic. Models achieved a peak accuracy of 0.86 in Medical Knowledge Question Answering, yet performance dipped significantly in Medical Language Understanding (0.71) and Complex Reasoning (0.72).

Furthermore, the study highlights a fragility in model robustness. While models scored 0.80 in safety and ethics accuracy, their robustness score dropped to 0.79 when options were shuffled. This suggests that models often rely on positional cues or "guessing" rather than a deep understanding of ethical constraints. In generative tasks, the problem shifts: 64% of errors were hallucinations, indicating that models struggle to maintain clinical accuracy when synthesizing information into a narrative format.

A Roadmap for Optimization

To bridge these gaps, the authors propose a four-tiered optimization strategy:

  • Level 1 (Surface): Enhancing performance through prompt engineering and parameter-efficient fine-tuning (e.g., LoRA).
  • Level 2 (Adaptation): Implementing Knowledge-Augmented Retrieval (RAG) and integrating explicit ethical constraints.
  • Level 3 (Architecture): Developing hybrid neuro-symbolic architectures that combine LLM linguistic fluidity with the reliable logic of symbolic AI.
  • Level 4 (Innovation): Advancing toward multimodal pre-training (text, imaging, genomics) and utilizing "digital twins" for safer, simulated clinical training.

By addressing these specific failure patterns, the medical AI community can move toward building tools that are not just knowledgeable, but truly capable of safe and robust clinical support.

Summary updated

Abstract

The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. Existing frameworks inadequately dissect domain-specific error patterns or address cross-modal challenges. This study introduces a granular error taxonomy through systematic analysis of top 10 models on MedBench, categorizing incorrect responses into eight types: Omissions, Hallucination, Format Mismatch, Causal Reasoning Deficiency, Contextual Inconsistency, Unanswered, Output Error, and Deficiency in Medical Language Generation. Evaluation of 10 leading models reveals vulnerabilities: despite achieving 0.86 accuracy in medical knowledge recall, critical reasoning tasks show 96.3% omission, while safety ethics evaluations expose alarming inconsistency (robustness score: 0.79) under option shuffled. Our analysis uncovers systemic weaknesses in knowledge boundary enforcement and multi-step reasoning. To address these, we propose a tiered optimization strategy spanning four levels, from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks. This work establishes an actionable roadmap for developing clinically robust LLMs while redefining evaluation paradigms through error-driven insights, ultimately advancing the safety and trustworthiness of AI in high-stakes medical environments.
BibTeX
@misc{jiang2025benchmarking,
  author = {Luyi Jiang and Jiayuan Chen and Lu Lu and Xinwei Peng and Lihao Liu and Junjun He and Jie Xu},
  title = {{Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies}},
  year = {2025},
  eprint = {2503.07306},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.07306},
}
Figure