GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians

Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan YangarXiv 2025 · DOI · arXiv

Introduction

Current medical AI benchmarks often rely on static, multiple-choice formats that prioritize rote memorization over clinical reasoning. These "medical trivia" tests fail to capture the nuances of real-world practice, where clinicians must manage uncertainty, provide comprehensive instructions, and maintain robustness against noisy or contradictory patient data. To address this, we introduce GAPS, a clinically grounded, automated benchmark designed to evaluate AI clinicians across four critical dimensions: Grounding, Adequacy, Perturbation, and Safety.

The GAPS Pipeline

The GAPS framework utilizes a multi-agent pipeline to transform authoritative clinical guidelines into structured, high-fidelity evaluation benchmarks. This process eliminates the need for slow, subjective manual rubric creation by human experts.

The GAPS automated pipeline for benchmark construction and scoring.

The pipeline operates in four distinct stages:

  1. Data Structuring: Guidelines (e.g., NCCN lung cancer protocols) are parsed into a Knowledge Graph and a Hierarchical Tree to capture logic and context.
  2. Item Generation: The system synthesizes questions with varying reasoning depths and introduces "perturbations" (linguistic noise or adversarial premises) to test model robustness.
  3. Rubric Synthesis: A "DeepResearch" agent uses a ReAct (Reasoning + Acting) loop and GRADE principles to extract "Must-have," "Should-have," and "Nice-to-have" criteria for every question.
  4. Automated Scoring: An ensemble of Large Language Models (LLMs) acts as a panel of judges to grade AI responses against these structured rubrics, incorporating a "safety floor" to flag dangerous recommendations.

Key Evaluation Axes

The benchmark evaluates models along four orthogonal axes:

  • Grounding (G): Measures reasoning depth, from simple factual recall (G1) to complex inference under clinical uncertainty (G4).
  • Adequacy (A): Assesses completeness, checking if the model includes necessary qualifiers and follow-up steps (A1-A3).
  • Perturbation (P): Tests robustness against linguistic noise or misleading adversarial premises (P1-P3).
  • Safety (S): Identifies risks ranging from harmless irrelevance to catastrophic "never events" (S1-S4).

Results and Model Performance

We evaluated state-of-the-art models, including GPT-5, Gemini 2.5 Pro, and Claude Opus 4, using the GAPS-NSCLC-preview dataset. This dataset comprises 92 complex questions and 1,691 individual rubric elements derived from NCCN guidelines for Non-Small Cell Lung Cancer.

Average model performance across different reasoning depths (G1-G4).

The "G2-G3 Cliff"

As shown in Figure 2, models demonstrate strong performance in factual recall (G1) and explanation (G2). However, a significant performance drop occurs at the G3 and G4 levels, where models must apply knowledge to specific patient vignettes or reason through "gray zones" of clinical uncertainty. GPT-5, while leading the cohort, saw scores drop from approximately 0.70 at G2 to below 0.45 at G4.

Adversarial Fragility and Safety

Models proved highly susceptible to P3 perturbations (adversarial premises). When prompts included subtle misinformation, models often failed to correct the premise and were "steered" into incorrect clinical conclusions. Furthermore, safety risks scaled with cognitive load; for instance, Claude Opus 4’s incidence of catastrophic (S4) errors increased from 3.3% at simple reasoning levels to 25% at the highest complexity level.

Conclusion

GAPS reveals that while modern LLMs are excellent medical encyclopedias, they struggle with the practical application of knowledge in complex, noisy clinical scenarios. The automated pipeline achieves a 90% agreement rate with human experts, providing a scalable path for rigorous AI clinician evaluation. Future work will expand the benchmark to include multi-turn dialogues, multimodal data, and longitudinal care tracking.

Summary updated

Abstract

Current benchmarks for AI clinician systems, often based on multiple-choice exams or manual rubrics, fail to capture the depth, robustness, and safety required for real-world clinical practice. To address this, we introduce the GAPS framework, a multidimensional paradigm for evaluating Grounding (cognitive depth), Adequacy (answer completeness), Perturbation (robustness), and Safety. Critically, we developed a fully automated, guideline-anchored pipeline to construct a GAPS-aligned benchmark end-to-end, overcoming the scalability and subjectivity limitations of prior work. Our pipeline assembles an evidence neighborhood, creates dual graph and tree representations, and automatically generates questions across G-levels. Rubrics are synthesized by a DeepResearch agent that mimics GRADE-consistent, PICO-driven evidence review in a ReAct loop. Scoring is performed by an ensemble of large language model (LLM) judges. Validation confirmed our automated questions are high-quality and align with clinician judgment (90% agreement, Cohen's Kappa 0.77). Evaluating state-of-the-art models on the benchmark revealed key failure modes: performance degrades sharply with increased reasoning depth (G-axis), models struggle with answer completeness (A-axis), and they are highly vulnerable to adversarial perturbations (P-axis) as well as certain safety issues (S-axis). This automated, clinically-grounded approach provides a reproducible and scalable method for rigorously evaluating AI clinician systems and guiding their development toward safer, more reliable clinical practice. The benchmark dataset GAPS-NSCLC-preview and evaluation code are publicly available at https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview and https://github.com/AQ-MedAI/MedicalAiBenchEval.
BibTeX
@misc{chen2025gaps,
  author = {Xiuyuan Chen and Tao Sun and Dexin Su and Ailing Yu and Junwei Liu and Zhe Chen and Gangzeng Jin and Xin Wang and Jingnan Liu and Hansong Xiao and Hualei Zhou and Dongjie Tao and Chunxiao Guo and Minghui Yang and Yuan Xia and Jing Zhao and Qianrui Fan and Yanyun Wang and Shuai Zhen and Kezhong Chen and Jun Wang and Zewen Sun and Heng Zhao and Tian Guan and Shaodong Wang and Geyun Chang and Jiaming Deng and Hongchengcheng Chen and Kexin Feng and Ruzhen Li and Jiayi Geng and Changtai Zhao and Jun Wang and Guihu Lin and Peihao Li and Liqi Liu and Peng Wei and Jian Wang and Jinjie Gu and Ping Wang and Fan Yang},
  title = {{GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians}},
  year = {2025},
  eprint = {2510.13734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.13734},
}
Figure