Introduction
HIV management represents a complex and evolving challenge in modern medicine, characterized by intricate drug regimens, diverse co-morbidities, and shifting resistance patterns. While large language models (LLMs) have demonstrated high proficiency in standardized multiple-choice examinations like the USMLE, these formats often fail to capture the nuances of real-world clinical decision-making. Researchers have introduced HIVMedQA, a specialized benchmark designed to evaluate LLMs in the context of "curbside consults"—the informal, open-ended clinical reasoning scenarios common in medical practice.
Methodology
The study utilizes a multi-layered evaluation framework to move beyond simple lexical accuracy:
- MedGPT Score (LLM-as-a-judge): GPT-4o acts as an automated evaluator, scoring responses on a 1-to-5 scale across five dimensions: reading comprehension, reasoning steps, knowledge recall, demographic bias, and potential for harm.
- MedSynF1: To address the limitations of traditional metrics like ROUGE or BLEU, the authors developed MedSynF1. This metric leverages the Unified Medical Language System (UMLS) and SNOMED CT to reward semantically correct medical terminology, even when it deviates from the exact phrasing of the "gold standard" answer.
- Prompt Engineering: Models were evaluated using a "senior physician" system prompt designed to prioritize brevity, clarity, and factual integrity.
The HIVQA Dataset
The benchmark is built upon the HIVQA dataset, which comprises four distinct categories validated by infectious disease specialists:
- Fundamental Knowledge: Basic diagnostic and factual questions.
- USMLE Step 1 Style: Standardized patient questions converted to open-ended formats.
- Complex Clinical Vignettes: Scenarios requiring multi-step reasoning and management.
- Biased Vignettes: Modified versions of Category 3 designed to test for cognitive biases such as recency and frequency bias.
Key Findings
The Reasoning Gap
A primary insight from the study is the persistent "reasoning gap." Models consistently performed better at knowledge recall than at complex clinical reasoning. While LLMs function as effective medical encyclopedias, their ability to synthesize information for patient management remains limited.
Generalists vs. Specialists
Counterintuitively, medically fine-tuned models (e.g., Meditron, Med42) did not consistently outperform general-purpose models. The researchers suggest that narrow fine-tuning may lead to overfitting or "catastrophic forgetting," whereas generalist models like Gemini 2.5 Pro benefit from broader logical training.
Susceptibility to Cognitive Bias
LLMs exhibited human-like cognitive biases. When presented with biased vignettes (Category 4), many models shifted their diagnoses toward irrelevant but recently mentioned information, demonstrating a susceptibility to recency bias.
Results and Performance
Gemini 2.5 Pro emerged as the top-performing model, leading in comprehension and reasoning across all categories. Claude 3.5 Sonnet v2 also showed strong performance. The study noted a significant gap between proprietary and open-source models, with proprietary giants generally dominating the leaderboard. Furthermore, the discrepancy between MedSynF1 scores (approx. 0.18) and MedGPT scores (4.0+) underscores that LLMs often provide clinically sound advice using natural language that varies significantly from structured reference texts.
Limitations and Future Directions
The current benchmark is limited to single-turn, text-only interactions. Future research will explore the integration of multimodal data (e.g., medical imaging) and the use of Retrieval-Augmented Generation (RAG) to determine if external clinical guidelines can help bridge the observed reasoning gap. Additionally, further human validation is needed to correlate MedGPT scores with expert physician assessments.
Summary updated