TL;DR
InclusionMed is a new open evaluation initiative that moves beyond "one-size-fits-all" medical AI leaderboards by mapping model performance across diverse healthcare roles and capabilities. By auditing 388 benchmarks down to a rigorous core set, the authors reveal that top-tier models often excel at communication but struggle with complex clinical reasoning and administrative agency—sometimes at a 24x higher inference cost than their peers.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) in medicine has outpaced our ability to evaluate them meaningfully. Current medical AI benchmarks are often fragmented, focusing heavily on multiple-choice medical exams that don't reflect real-world clinical workflows. Furthermore, evaluation is rarely "inclusive"; it often ignores the needs of patients and administrative staff, focuses on English-speaking Western contexts, and fails to account for the massive variance in inference costs.
The InclusionMed Collaboration tackles this by introducing a structured framework to organize medical AI evaluation, ensuring that "progress" is measured against the actual needs of the global healthcare ecosystem.
Methods
The authors developed the InclusionMed Atlas, a 3x3 matrix that categorizes benchmarks by Healthcare Role (Patients, Professionals, Administrators) and AI Capability (Knowledge, Reasoning, Agency).
To populate this Atlas, they performed a massive "audit" of 388 candidate benchmarks using six criteria:
- Healthcare Relevance: Does it address real-world needs?
- Trustworthiness: Is the ground truth medically sound (and not just AI-generated)?
- Access & Use: Are the tasks and scoring code actually available?
- Complementarity: Does it add new value to the portfolio?
- Timeliness & Independence: Is the data leaked or outdated?
- Discriminative Capacity: Can it actually tell the difference between top models, or is everyone scoring 95%+?
After this rigorous filtering and independent reproduction of results, only five benchmarks were retained for the initial InclusionMed Index.
Datasets
The final Index is composed of five benchmarks, each weighted at 20%:
- HealthBench Professional: Evaluates reasoning in clinician-model conversations.
- PhysicianBench: Tests "agency" by having models interact with a simulated Electronic Health Record (EHR) environment.
- ALE-CLI v1.0 (Health and Medicine): Focuses on computational tasks in medical research.
- CHI-Bench: Evaluates administrative agency (e.g., prior authorization and care management).
- InclusionMed-DxFiles (New): A private benchmark introduced in this paper. It uses 312 complex cases from the New England Journal of Medicine to test diagnostic reasoning where the final diagnosis is withheld.
Key Insights and Ideas
- Leaderboards are Context-Specific: A model's rank changes depending on the role. For example, while Gemini 4 Argon ranked 1st overall, Claude Opus 5.5 led for healthcare professionals, and GPT-6 Astra was the top performer in pure reasoning.
- The "Communication-Reasoning" Gap: The ten highest-ranked models all scored significantly higher on patient communication than on clinical decision-making. Models are "polite" but still struggle with the hard logic of treatment planning.
- Benchmark Saturation is Real: The authors had to exclude several popular benchmarks (like NEJMQA-Exams) because frontier models have reached "saturation"—the scores are so high and tightly packed that the benchmarks can no longer distinguish between a good model and a great one.
- The Cost of "Intelligence": Models with nearly identical performance scores can have wildly different price tags. The authors found an approximately 24-fold difference in mean cost per task between certain models on the Pareto frontier.
Key Results
The team evaluated 22 models from 11 families. Overall Index scores ranged from a low of 17.5 to a high of 49.1 out of 100, suggesting that even "frontier" models have a long way to go in complex medical agency.
One of the most practical outputs is the Performance-Cost Trade-off analysis. By plotting the Index score against estimated inference cost, they identified a Pareto frontier of models that offer the best "bang for the buck," including GPT-6 Luna at the low-cost end and Gemini 4 Argon at the high-performance end.
Limitations and Future Work
The authors are transparent about the "gaps" in their own Atlas:
- Empty Cells: There are currently no eligible benchmarks for "Patients and the Public" that passed the audit, highlighting a major blind spot in the research community.
- Geographic Bias: 94% of tasks with a determinable setting were associated with the United States.
- Modality: 99.8% of the tasks were text-only, ignoring the critical role of medical imaging and waveforms.
- Future Directions: The collaboration is calling for community contributions to fill these gaps, particularly for non-English languages and multi-modal diagnostic tasks.
Summary updated