Introduction
Current medical AI benchmarks often rely on static, multiple-choice formats that prioritize rote memorization over clinical reasoning. These "medical trivia" tests fail to capture the nuances of real-world practice, where clinicians must manage uncertainty, provide comprehensive instructions, and maintain robustness against noisy or contradictory patient data. To address this, we introduce GAPS, a clinically grounded, automated benchmark designed to evaluate AI clinicians across four critical dimensions: Grounding, Adequacy, Perturbation, and Safety.
The GAPS Pipeline
The GAPS framework utilizes a multi-agent pipeline to transform authoritative clinical guidelines into structured, high-fidelity evaluation benchmarks. This process eliminates the need for slow, subjective manual rubric creation by human experts.
The pipeline operates in four distinct stages:
- Data Structuring: Guidelines (e.g., NCCN lung cancer protocols) are parsed into a Knowledge Graph and a Hierarchical Tree to capture logic and context.
- Item Generation: The system synthesizes questions with varying reasoning depths and introduces "perturbations" (linguistic noise or adversarial premises) to test model robustness.
- Rubric Synthesis: A "DeepResearch" agent uses a ReAct (Reasoning + Acting) loop and GRADE principles to extract "Must-have," "Should-have," and "Nice-to-have" criteria for every question.
- Automated Scoring: An ensemble of Large Language Models (LLMs) acts as a panel of judges to grade AI responses against these structured rubrics, incorporating a "safety floor" to flag dangerous recommendations.
Key Evaluation Axes
The benchmark evaluates models along four orthogonal axes:
- Grounding (G): Measures reasoning depth, from simple factual recall (G1) to complex inference under clinical uncertainty (G4).
- Adequacy (A): Assesses completeness, checking if the model includes necessary qualifiers and follow-up steps (A1-A3).
- Perturbation (P): Tests robustness against linguistic noise or misleading adversarial premises (P1-P3).
- Safety (S): Identifies risks ranging from harmless irrelevance to catastrophic "never events" (S1-S4).
Results and Model Performance
We evaluated state-of-the-art models, including GPT-5, Gemini 2.5 Pro, and Claude Opus 4, using the GAPS-NSCLC-preview dataset. This dataset comprises 92 complex questions and 1,691 individual rubric elements derived from NCCN guidelines for Non-Small Cell Lung Cancer.
The "G2-G3 Cliff"
As shown in Figure 2, models demonstrate strong performance in factual recall (G1) and explanation (G2). However, a significant performance drop occurs at the G3 and G4 levels, where models must apply knowledge to specific patient vignettes or reason through "gray zones" of clinical uncertainty. GPT-5, while leading the cohort, saw scores drop from approximately 0.70 at G2 to below 0.45 at G4.
Adversarial Fragility and Safety
Models proved highly susceptible to P3 perturbations (adversarial premises). When prompts included subtle misinformation, models often failed to correct the premise and were "steered" into incorrect clinical conclusions. Furthermore, safety risks scaled with cognitive load; for instance, Claude Opus 4’s incidence of catastrophic (S4) errors increased from 3.3% at simple reasoning levels to 25% at the highest complexity level.
Conclusion
GAPS reveals that while modern LLMs are excellent medical encyclopedias, they struggle with the practical application of knowledge in complex, noisy clinical scenarios. The automated pipeline achieves a 90% agreement rate with human experts, providing a scalable path for rigorous AI clinician evaluation. Future work will expand the benchmark to include multi-turn dialogues, multimodal data, and longitudinal care tracking.
Summary updated