Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation

Xiangxu Zhang, Lei Li, Yanyun Zhou, Xiao Zhou, Yingying Zhang, Xian Wu

Introduction: The Clinical Gap

Large Language Models (LLMs) have demonstrated remarkable proficiency in medical licensing exams, often achieving scores that rival or exceed human experts. However, these standardized benchmarks often fail to reflect the complexities of real-world clinical practice. A doctor’s office is not a multiple-choice test; patients rarely use precise clinical terminology like "nocturia," instead opting for colloquialisms such as "I wake up to pee a lot."

Furthermore, current evaluations suffer from two critical flaws. First, data contamination is a growing concern; because static benchmarks are public, they frequently leak into LLM training sets, allowing models to "memorize" answers rather than reason through cases. Second, there is a significant clinical misalignment. Standardized tests are "clean," whereas real clinical practice is "noisy," involving ambiguous symptoms, distracting medical histories, and patient biases.

Comparison of standardized exam questions versus noisy real-world patient descriptions.

The DyReMe Framework

To address these limitations, this research introduces DyReMe (Dynamic, Realistic evaluation for Medical diagnostics). DyReMe moves beyond static shuffling of questions to generate "messy," realistic clinical consultations that stress-test a model's true diagnostic capabilities.

1. DyGen: Dynamic Question Generation

Instead of simple paraphrasing, the DyGen module uses Retrieval-Augmented Generation (RAG) to identify "differential diagnoses"—diseases with symptoms similar to the target illness. It then weaves "diagnostic traps" into the prompts:

  • Self-Diagnosis: The simulated patient insists on an incorrect condition they found online.
  • Distracting History: Irrelevant medical background is added to obscure clinical clues.
  • Expression Style: Prompts are rewritten using various "personas" (e.g., low medical literacy) to test if the model can still extract relevant symptoms.
  • Validator-Refiner Loop: An iterative process ensures generated questions remain medically sound and logically consistent.

2. EvalMed: Multi-Dimensional Evaluation

Moving beyond simple accuracy, the EvalMed module scores models across four dimensions:

  • Accuracy: Correctness of the primary diagnosis.
  • Veracity: The model's ability to identify and correct medical rumors or misinformation planted in the query.
  • Helpfulness: Whether the response provides actionable advice regarding evidence, treatment, and lifestyle.
  • Consistency: Whether the model maintains the same diagnosis when the same case is described in different styles or personas.
The DyReMe pipeline: (a) Differential Diagnosis generation, (b) DyGen question generation with traps, and (c) EvalMed multi-dimensional evaluation.

Experimental Results and Key Insights

The researchers utilized three established datasets—DxBench, DDXPlus, and Dxy—as "seeds" to generate a challenging benchmark of 3,200 dynamic questions. Evaluating 12 leading LLMs revealed several critical findings:

  • The "Trap" Effect: Models are highly susceptible to confirmation bias. When a simulated patient suggests a wrong diagnosis, many LLMs follow the patient's lead rather than correcting them based on the provided symptoms.
  • Significant Performance Drop: Accuracy plummeted across the board when moving from static benchmarks to DyReMe. For instance, DeepSeek-V3 saw a relative drop of 10.51%, and GPT-4o dropped by 10.75%.
  • The Veracity Gap: Even top-tier models failed to correct medical misinformation 20–40% of the time, highlighting a dangerous lack of robustness in clinical reasoning.
  • Consistency as an Achilles' Heel: Many models proved "brittle." Changing a patient's description of pain from "radiating" to "like an electric shock" frequently caused models to change their entire diagnosis.

Conclusion

The DyReMe framework demonstrates that current medical LLM performance may be inflated by static, clean benchmarks. While commercial models like GPT-4o and Qwen-Max currently lead in handling clinical noise, significant gaps remain in veracity and consistency. Future work will look toward multimodal diagnostics—incorporating lab reports and imaging—to further bridge the gap between AI evaluation and clinical reality.

Summary updated

BibTeX
@inproceedings{zhang2026inflated,
  author = {Xiangxu Zhang and Lei Li and Yanyun Zhou and Xiao Zhou and Yingying Zhang and Xian Wu},
  title = {{Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1218},
}
Figure