How to Evaluate Medical AI

Ilia Kopanichuk, Petr Anokhin, Vladimir Shaposhnikov, Vladimir Makharev, Ekaterina Tsapieva, Iaroslav Bespalov, Dmitry V. Dylov, Ivan V. OseledetsarXiv 2025 · DOI · arXiv

Abstract

Evaluating medical AI is notoriously difficult because human doctors often disagree on the "correct" diagnosis. This paper introduces RPAD and RRAD, new relative metrics that evaluate AI by comparing its performance to the level of disagreement among a panel of experts, rather than a single "ground truth." Using these metrics, the authors found that top-tier models like DeepSeek-V3 can match or even exceed the diagnostic consistency of human physicians.

The Challenge of Medical Benchmarking

In most machine learning tasks, a "gold standard" label provides a clear target. In medicine, such a standard rarely exists. If the same patient case is presented to five different doctors, it is common to receive multiple distinct diagnoses. Traditional metrics like Precision and Recall fail in this environment because they assume one expert is 100% correct while all others are wrong.

Furthermore, real-world clinical practice relies on free-form text rather than pre-defined disease lists. Comparing a term like "acute viral rhinopharyngitis" to "common cold" requires a deep understanding of clinical synonymy. This research addresses both the lack of a stable benchmark and the complexity of free-form diagnostic matching.

Methodology

The researchers proposed a two-pillar approach to modernize medical AI evaluation:

1. Relative Metrics (RPAD & RRAD)

Instead of absolute scores, the study introduces the Relative Precision of Algorithmic Diagnostics (RPAD) and Relative Recall of Algorithmic Diagnostics (RRAD). These metrics compare the AI’s agreement with a panel of experts against the experts’ agreement with one another.

  • A score of 1.0 indicates the AI is as consistent with the experts as they are with themselves.
  • A score > 1.0 suggests the AI is closer to the "average" expert opinion than the most deviant human expert is.

2. The Match Function (M)

To compare free-form text diagnoses, the authors developed a supervised meta-model (a bagged ensemble of LightGBM) utilizing 17 distinct features to determine if two strings represent the same clinical diagnosis. These features include direct LLM prompting, semantic similarity via text embeddings (Giga, BERGAMOT, Arctic), and clinical grounding through a RAG system mapped to the ICD-10 dictionary.

Correlation analysis of the features integrated into the diagnostic match function.

Key Insights

  • Expert Variance is Significant: The study confirmed that human experts exhibit high variability. In many instances, the disagreement between two human physicians was greater than the disagreement between the AI and the human panel.
  • Normalizing for Difficulty: The authors introduced a "hardness parameter" ($H$) in their formulas, allowing evaluators to shift from an "optimistic" view (best-case agreement) to an "average" view for a more nuanced look at model reliability.
  • Automated Matching is Feasible: By combining LLMs with traditional NLP and clinical dictionaries, the match function achieved 98% accuracy in identifying functionally identical clinical diagnoses.

Results and Performance

DeepSeek-V3 emerged as the top performer, achieving an RPAD@1 of 1.15 and an RRAD@3 of 1.16. This indicates that the model is more consistent with the expert consensus than the experts are with each other. In contrast, models like Dist-Qwen32B fell below the 1.0 threshold, suggesting they have not yet reached parity with human expert panels in this diagnostic context.

RPAD and RRAD scores for DeepSeek-V3 and Dist-Qwen32B. The dashed line at 1.0 represents the human expert baseline.

Conclusion and Future Directions

While the results for top-tier models are promising, the high variance among human experts suggests a need for clearer diagnostic guidelines to create a more stable baseline. Future work will involve expanding the dataset beyond the initial 360 cases to include more diverse and rare medical scenarios. Additionally, the authors aim to incorporate clinical risk weighting into the metrics to account for the severity of potential misdiagnoses.

Summary updated

BibTeX
@misc{kopanichuk2025how,
  author = {Ilia Kopanichuk and Petr Anokhin and Vladimir Shaposhnikov and Vladimir Makharev and Ekaterina Tsapieva and Iaroslav Bespalov and Dmitry V. Dylov and Ivan V. Oseledets},
  title = {{How to Evaluate Medical AI}},
  year = {2025},
  eprint = {2509.11941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.11941},
}
Figure