HIVMedQA: Benchmarking large language models for HIV medical decision support

Gonzalo Cardenal-Antolin, Jacques Fellay, Bashkim Jaha, Roger D. Kouyos, Niko Beerenwinkel, Diane DurouxarXiv 2025 · DOI · arXiv

Introduction

HIV management represents a complex and evolving challenge in modern medicine, characterized by intricate drug regimens, diverse co-morbidities, and shifting resistance patterns. While large language models (LLMs) have demonstrated high proficiency in standardized multiple-choice examinations like the USMLE, these formats often fail to capture the nuances of real-world clinical decision-making. Researchers have introduced HIVMedQA, a specialized benchmark designed to evaluate LLMs in the context of "curbside consults"—the informal, open-ended clinical reasoning scenarios common in medical practice.

Methodology

The study utilizes a multi-layered evaluation framework to move beyond simple lexical accuracy:

  • MedGPT Score (LLM-as-a-judge): GPT-4o acts as an automated evaluator, scoring responses on a 1-to-5 scale across five dimensions: reading comprehension, reasoning steps, knowledge recall, demographic bias, and potential for harm.
  • MedSynF1: To address the limitations of traditional metrics like ROUGE or BLEU, the authors developed MedSynF1. This metric leverages the Unified Medical Language System (UMLS) and SNOMED CT to reward semantically correct medical terminology, even when it deviates from the exact phrasing of the "gold standard" answer.
  • Prompt Engineering: Models were evaluated using a "senior physician" system prompt designed to prioritize brevity, clarity, and factual integrity.

The HIVQA Dataset

The benchmark is built upon the HIVQA dataset, which comprises four distinct categories validated by infectious disease specialists:

  1. Fundamental Knowledge: Basic diagnostic and factual questions.
  2. USMLE Step 1 Style: Standardized patient questions converted to open-ended formats.
  3. Complex Clinical Vignettes: Scenarios requiring multi-step reasoning and management.
  4. Biased Vignettes: Modified versions of Category 3 designed to test for cognitive biases such as recency and frequency bias.
Performance of various LLMs across the first three categories of the HIVQA dataset, showing MedGPT scores for comprehension, reasoning, and recall.

Key Findings

The Reasoning Gap

A primary insight from the study is the persistent "reasoning gap." Models consistently performed better at knowledge recall than at complex clinical reasoning. While LLMs function as effective medical encyclopedias, their ability to synthesize information for patient management remains limited.

Generalists vs. Specialists

Counterintuitively, medically fine-tuned models (e.g., Meditron, Med42) did not consistently outperform general-purpose models. The researchers suggest that narrow fine-tuning may lead to overfitting or "catastrophic forgetting," whereas generalist models like Gemini 2.5 Pro benefit from broader logical training.

Susceptibility to Cognitive Bias

LLMs exhibited human-like cognitive biases. When presented with biased vignettes (Category 4), many models shifted their diagnoses toward irrelevant but recently mentioned information, demonstrating a susceptibility to recency bias.

Radar chart and line graph illustrating the performance drop between Category 3 (complex vignettes) and Category 4 (biased vignettes), highlighting the impact of cognitive bias on model reasoning.

Results and Performance

Gemini 2.5 Pro emerged as the top-performing model, leading in comprehension and reasoning across all categories. Claude 3.5 Sonnet v2 also showed strong performance. The study noted a significant gap between proprietary and open-source models, with proprietary giants generally dominating the leaderboard. Furthermore, the discrepancy between MedSynF1 scores (approx. 0.18) and MedGPT scores (4.0+) underscores that LLMs often provide clinically sound advice using natural language that varies significantly from structured reference texts.

Limitations and Future Directions

The current benchmark is limited to single-turn, text-only interactions. Future research will explore the integration of multimodal data (e.g., medical imaging) and the use of Retrieval-Augmented Generation (RAG) to determine if external clinical guidelines can help bridge the observed reasoning gap. Additionally, further human validation is needed to correlate MedGPT scores with expert physician assessments.

Summary updated

Abstract

Large language models (LLMs) are emerging as valuable tools to support clinicians in routine decision-making. HIV management is a compelling use case due to its complexity, including diverse treatment options, comorbidities, and adherence challenges. However, integrating LLMs into clinical practice raises concerns about accuracy, potential harm, and clinician acceptance. Despite their promise, AI applications in HIV care remain underexplored, and LLM benchmarking studies are scarce. This study evaluates the current capabilities of LLMs in HIV management, highlighting their strengths and limitations. We introduce HIVMedQA, a benchmark designed to assess open-ended medical question answering in HIV care. The dataset consists of curated, clinically relevant questions developed with input from an infectious disease physician. We evaluated seven general-purpose and three medically specialized LLMs, applying prompt engineering to enhance performance. Our evaluation framework incorporates both lexical similarity and an LLM-as-a-judge approach, extended to better reflect clinical relevance. We assessed performance across key dimensions: question comprehension, reasoning, knowledge recall, bias, potential harm, and factual accuracy. Results show that Gemini 2.5 Pro consistently outperformed other models across most dimensions. Notably, two of the top three models were proprietary. Performance declined as question complexity increased. Medically fine-tuned models did not always outperform general-purpose ones, and larger model size was not a reliable predictor of performance. Reasoning and comprehension were more challenging than factual recall, and cognitive biases such as recency and status quo were observed. These findings underscore the need for targeted development and evaluation to ensure safe, effective LLM integration in clinical care.
BibTeX
@misc{cardenalantolin2025hivmedqa,
  author = {Gonzalo Cardenal-Antolin and Jacques Fellay and Bashkim Jaha and Roger D. Kouyos and Niko Beerenwinkel and Diane Duroux},
  title = {{HIVMedQA: Benchmarking large language models for HIV medical decision support}},
  year = {2025},
  eprint = {2507.18143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.18143},
}
Figure