InclusionMed: Benchmarking AI for Inclusive Healthcare

The InclusionMed CollaborationarXiv 2026 · arXiv

TL;DR

InclusionMed is a new open evaluation initiative that moves beyond "one-size-fits-all" medical AI leaderboards by mapping model performance across diverse healthcare roles and capabilities. By auditing 388 benchmarks down to a rigorous core set, the authors reveal that top-tier models often excel at communication but struggle with complex clinical reasoning and administrative agency—sometimes at a 24x higher inference cost than their peers.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) in medicine has outpaced our ability to evaluate them meaningfully. Current medical AI benchmarks are often fragmented, focusing heavily on multiple-choice medical exams that don't reflect real-world clinical workflows. Furthermore, evaluation is rarely "inclusive"; it often ignores the needs of patients and administrative staff, focuses on English-speaking Western contexts, and fails to account for the massive variance in inference costs.

The InclusionMed Collaboration tackles this by introducing a structured framework to organize medical AI evaluation, ensuring that "progress" is measured against the actual needs of the global healthcare ecosystem.

Methods

The authors developed the InclusionMed Atlas, a 3x3 matrix that categorizes benchmarks by Healthcare Role (Patients, Professionals, Administrators) and AI Capability (Knowledge, Reasoning, Agency).

The InclusionMed Atlas organizes benchmarks into a role-capability matrix.

To populate this Atlas, they performed a massive "audit" of 388 candidate benchmarks using six criteria:

  1. Healthcare Relevance: Does it address real-world needs?
  2. Trustworthiness: Is the ground truth medically sound (and not just AI-generated)?
  3. Access & Use: Are the tasks and scoring code actually available?
  4. Complementarity: Does it add new value to the portfolio?
  5. Timeliness & Independence: Is the data leaked or outdated?
  6. Discriminative Capacity: Can it actually tell the difference between top models, or is everyone scoring 95%+?

After this rigorous filtering and independent reproduction of results, only five benchmarks were retained for the initial InclusionMed Index.

Datasets

The final Index is composed of five benchmarks, each weighted at 20%:

  • HealthBench Professional: Evaluates reasoning in clinician-model conversations.
  • PhysicianBench: Tests "agency" by having models interact with a simulated Electronic Health Record (EHR) environment.
  • ALE-CLI v1.0 (Health and Medicine): Focuses on computational tasks in medical research.
  • CHI-Bench: Evaluates administrative agency (e.g., prior authorization and care management).
  • InclusionMed-DxFiles (New): A private benchmark introduced in this paper. It uses 312 complex cases from the New England Journal of Medicine to test diagnostic reasoning where the final diagnosis is withheld.

Key Insights and Ideas

  • Leaderboards are Context-Specific: A model's rank changes depending on the role. For example, while Gemini 4 Argon ranked 1st overall, Claude Opus 5.5 led for healthcare professionals, and GPT-6 Astra was the top performer in pure reasoning.
  • The "Communication-Reasoning" Gap: The ten highest-ranked models all scored significantly higher on patient communication than on clinical decision-making. Models are "polite" but still struggle with the hard logic of treatment planning.
  • Benchmark Saturation is Real: The authors had to exclude several popular benchmarks (like NEJMQA-Exams) because frontier models have reached "saturation"—the scores are so high and tightly packed that the benchmarks can no longer distinguish between a good model and a great one.
  • The Cost of "Intelligence": Models with nearly identical performance scores can have wildly different price tags. The authors found an approximately 24-fold difference in mean cost per task between certain models on the Pareto frontier.

Key Results

The team evaluated 22 models from 11 families. Overall Index scores ranged from a low of 17.5 to a high of 49.1 out of 100, suggesting that even "frontier" models have a long way to go in complex medical agency.

InclusionMed Index scores for 22 models, showing Gemini 4 Argon and the Claude 5 series in the lead.

One of the most practical outputs is the Performance-Cost Trade-off analysis. By plotting the Index score against estimated inference cost, they identified a Pareto frontier of models that offer the best "bang for the buck," including GPT-6 Luna at the low-cost end and Gemini 4 Argon at the high-performance end.

The Pareto frontier of performance vs. cost.

Limitations and Future Work

The authors are transparent about the "gaps" in their own Atlas:

  • Empty Cells: There are currently no eligible benchmarks for "Patients and the Public" that passed the audit, highlighting a major blind spot in the research community.
  • Geographic Bias: 94% of tasks with a determinable setting were associated with the United States.
  • Modality: 99.8% of the tasks were text-only, ignoring the critical role of medical imaging and waveforms.
  • Future Directions: The collaboration is calling for community contributions to fill these gaps, particularly for non-English languages and multi-modal diagnostic tasks.

Summary updated

Abstract

Advancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, but benchmark coverage remains uneven. We introduce InclusionMed, an open medical AI evaluation initiative for inclusive healthcare. The InclusionMed Atlas organizes benchmarks across three healthcare roles and three AI capabilities. For the first release, we selected four public benchmarks from 388 candidates through structured audits and independent reproduction. With InclusionMed-DxFiles, our private diagnostic reasoning benchmark, the Atlas contains five benchmarks covering three of its nine role--capability cells. We evaluated frontier large language models on the Atlas and aggregated their results into the InclusionMed Index, with overall scores ranging from 17.5 to 49.1 out of 100. The Index also reports scores by healthcare role and AI capability, showing that a model's rank varied with the role or capability considered. A within-benchmark analysis using task-level tags showed that the ten highest-ranked models in the Index all scored higher on patient communication than on clinical decision-making. Models with similar Index scores also differed substantially in estimated inference cost. These findings highlight the need for evaluation covering more diverse healthcare needs and for context-specific model selection. Through open collaboration and community contributions, InclusionMed seeks to ensure that diverse healthcare needs worldwide shape how medical AI progress is defined and measured.
BibTeX
@misc{collaboration2026inclusionmed,
  author = {The InclusionMed Collaboration},
  title = {{InclusionMed: Benchmarking AI for Inclusive Healthcare}},
  year = {2026},
  eprint = {2611.21218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21218},
}
Figure