Beyond Overall Accuracy: A Psychometric Deep Dive into the Topic-Specific Medical Capabilities of 80 Large Language Models

He Daqing, Zhimeng Luo, Lixin Wu, Adam Frisch, Daqing HearXiv 2025 · DOI · arXiv

Abstract

Traditional evaluation of medical Large Language Models (LLMs) relies heavily on aggregate accuracy, a metric that often masks significant variations in domain-specific performance. This paper introduces MEDIRT, a framework applying Item Response Theory (IRT) to evaluate 80 LLMs on a USMLE-aligned benchmark. Our analysis reveals that even top-tier models exhibit "spiky" ability profiles—excelling in certain medical specialties while underperforming in others. By shifting from accuracy to latent ability ($θ$), we provide a more nuanced understanding of model competence, identify flawed benchmark questions, and highlight the Pareto frontier of cost-performance trade-offs in clinical AI.

Introduction: The Limitations of Aggregate Accuracy

In high-stakes medical applications, evaluating LLMs based solely on the percentage of correct answers is insufficient. Aggregate accuracy suffers from two primary flaws: item indifference and the "spiky" profile mask. It treats simple recall questions and complex diagnostic puzzles as equal, allowing a model to achieve a high score by mastering easy items while failing difficult ones. Furthermore, a single score obscures a model's specific strengths and weaknesses; a model might be highly proficient in cardiology but dangerously incompetent in medical ethics, yet both are averaged into one misleading number.

To deploy LLMs safely in healthcare, we must measure latent ability—a statistical measure of competence that accounts for question difficulty and topic-specific nuances.

The MEDIRT Framework

The researchers proposed MEDIRT, a framework grounded in Item Response Theory (IRT), the gold standard for high-stakes human testing like the USMLE or GRE.

The MedIRT Framework: A three-pillar approach combining psychometric rigor, USMLE-aligned benchmarking, and large-scale LLM evaluation.

Methodology

  • The 2PL Model: A two-parameter logistic IRT model was used to estimate two variables for every question: Difficulty ($b$) and Discrimination ($a$) (the question's ability to distinguish between high- and low-ability models).
  • Latent Ability ($θ$): Models are ranked by $θ$, a standardized score where 0 represents the cohort average.
  • Topic-Specific Modeling: To prevent "multidimensionality" issues, 11 independent IRT models were fitted for specific medical topics (e.g., Cardiovascular, Gastrointestinal, Social Science).
  • Benchmark Alignment: The study utilized a 1,100-question benchmark sampled from MedQA, MedMCQA, and MedXpertQA, balanced to align with USMLE Step 1 Content Specifications.

Results: The "Spiky" Reality of Medical LLMs

The most striking finding is that model performance is rarely uniform across domains. While GPT-5 led the cohort overall, it exhibited significant variance across specialties. For instance, it was outperformed in "Social Science and Communication" by Claude-3-opus, despite the latter ranking 23rd in overall accuracy.

Topic-wise IRT Ability (left) vs. Accuracy (right) for the top 5 models. The radar charts reveal 'spiky' profiles where models excel in specific domains while lagging in others.

Rank Flips and Benchmark Auditing

Accuracy and IRT Ability do not always correlate perfectly. GPT-4o improved its relative ranking when measured by IRT because it consistently solved more difficult, high-discrimination questions. Conversely, some models dropped in rank, revealing that their accuracy was "inflated" by easier questions.

IRT also served as a diagnostic tool for the benchmark itself. The authors identified 15 questions with negative discrimination, where high-ability models were statistically more likely to fail. Investigation revealed "design pathologies," such as flawed "All of the above" options or imprecise distractors that penalized more sophisticated reasoning.

Discussion: Efficiency and Future Directions

By mapping latent ability against cost and latency, the study identified "efficiency leaders." GPT-oss-120b emerged as a powerful open-source alternative, offering high ability at a fraction of the cost of proprietary frontier models.

Despite these advances, limitations remain. The current multiple-choice format cannot fully capture the conversational nature of clinical reasoning, and the topic-independent modeling does not yet account for the interdisciplinary integration required in real-world medicine. Future work will focus on merging disparate benchmarks into a unified "universal scale" for medical AI evaluation.

Conclusion

Moving beyond the scoreboard is essential for the responsible integration of AI in medicine. The MEDIRT framework demonstrates that psychometric analysis provides the granularity needed to identify model-specific risks and benchmark flaws that simple accuracy metrics overlook.

Summary updated

BibTeX
@misc{daqing2025beyond,
  author = {He Daqing and Zhimeng Luo and Lixin Wu and Adam Frisch and Daqing He},
  title = {{Beyond Overall Accuracy: A Psychometric Deep Dive into the Topic-Specific Medical Capabilities of 80 Large Language Models}},
  year = {2025},
  eprint = {2509.24186},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24186},
}
Figure