Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Shefayat E. Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham, Ajwad Abrar, Tareque Mohmud ChowdhuryarXiv 2026 · arXiv · OpenReview

TL;DR

This paper benchmarks five recent Large Language Models (LLMs) on medical question answering using a zero-shot approach and an LLM-as-a-Judge framework. The study reveals that larger models like Llama 3.3 70B Instruct perform best, but architectural innovations in models like Llama-4 Maverick 17B offer competitive efficiency, highlighting the trade-offs for real-world deployment.

What Problem Does This Paper Address?

The increasing adoption of LLMs in healthcare, particularly for medical question answering (QA), necessitates rigorous evaluation to ensure accuracy, safety, and clinical utility. While LLMs show promise, they often struggle with specialized medical knowledge depth and can exhibit issues like hallucinations and inconsistent reasoning, especially in real-world clinical scenarios. Existing evaluation methods, often relying on multiple-choice exams, may not fully capture the nuances of generative medical QA. This paper addresses the need for a comprehensive evaluation of contemporary LLMs in medical QA, comparing their performance across diverse specialties and exploring the impact of model size and architecture.

Methods

The authors employ a two-pronged evaluation strategy:

  1. Zero-Shot Evaluation with Traditional Metrics: They use a zero-shot learning approach, meaning the LLMs are prompted to answer medical questions without any task-specific fine-tuning or in-context examples. This assesses their inherent medical knowledge acquired during pre-training. Performance is measured using standard automatic metrics:
  2. BLEU (Bilingual Evaluation Understudy): Measures n-gram overlap between generated and reference answers, focusing on precision.
  3. ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Evaluates recall-oriented overlap, assessing how well generated answers capture essential information from reference responses (ROUGE-1, ROUGE-2, ROUGE-L).
  4. LLM-as-a-Judge Framework: To overcome the limitations of n-gram metrics in assessing factual correctness, safety, and clinical utility, they implement an LLM-as-a-Judge framework. Claude Sonnet 4 is used as the "judge" model to evaluate responses across five key dimensions on a 5-point Likert scale:
  5. Medical Accuracy: Factual correctness and appropriate use of qualifiers.
  6. Completeness: Adequacy of addressing all aspects of the question.
  7. Safety: Appropriateness of disclaimers and avoidance of harmful advice.
  8. Clarity: Understandability of the communication.
  9. Helpfulness: Practical utility and actionable advice. A weighted score is calculated, prioritizing Medical Accuracy (30%) and Safety (25%).

Datasets

  • iCliniq Medical QA Dataset: This dataset, containing 38,000 real-world medical questions and expert-validated answers across various specialties, forms the basis of the evaluation. For computational feasibility, a subset of 3,000 question-answer pairs was randomly selected, maintaining the original distribution of medical specialties.
  • MedLM Baselines: For comparative context, the paper references baseline results from prior work (MedLM) using models like GPT-2, T5, and GPT-3.5 on the same iCliniq dataset.

Key Insights and Ideas

  • Scaling Still Matters, But Architecture is Catching Up: While larger models like Llama 3.3 70B Instruct generally outperform smaller ones, the study highlights that architectural innovations, such as those in Llama-4 Maverick 17B, can significantly close the performance gap with fewer parameters. This suggests a potential shift towards more efficient, yet powerful, models.
  • Zero-Shot is a Strong Baseline, But Not the Whole Story: The zero-shot evaluation provides a valuable assessment of LLMs' inherent capabilities. However, the LLM-as-a-Judge framework reveals that traditional metrics like BLEU and ROUGE alone are insufficient for evaluating critical aspects like safety and factual accuracy in medical contexts.
  • Medical QA Demands Multi-Dimensional Evaluation: The LLM-as-a-Judge results underscore the importance of considering multiple facets of response quality beyond simple lexical overlap. Safety and accuracy are paramount, and models that excel in these areas, even with lower traditional scores, may be more suitable for clinical applications.
  • Efficiency Trade-offs are Crucial for Deployment: The paper emphasizes that models like Llama-4 Maverick 17B, offering high performance with significantly fewer parameters than Llama 3.3 70B, present attractive options for resource-constrained clinical settings. This highlights the practical considerations for deploying LLMs in real-world healthcare.

Key Results

  • Top Performer: Llama 3.3 70B Instruct achieved the highest scores across traditional metrics (BLEU-1: 0.2207, ROUGE-1: 0.2761, ROUGE-L: 0.1306) and the LLM-as-a-Judge evaluation (Overall Score: 4.40/5.00).
  • Efficient Contender: Llama-4 Maverick 17B demonstrated competitive performance, ranking second in most metrics and achieving the highest clarity score (4.97/5) in the LLM-as-a-Judge evaluation, despite having fewer parameters.
  • Underperformer: GPT-5-mini showed suboptimal performance across all metrics, with a BLEU-1 score of 0.0124 and the lowest ROUGE-L score (0.0914).
  • Significant Improvement Over Baselines: The evaluated Llama models showed substantial improvements over older baseline models (GPT-2, T5, GPT-3.5), with Llama 3.3 70B achieving a 2.2x improvement in BLEU-1 and a 125x improvement in ROUGE-1 compared to the strongest MedLM baseline.
  • Correlation: ROUGE-1 showed a strong positive correlation (r=0.89) with overall judge scores, while BLEU metrics had a moderate correlation (r=0.67) with medical accuracy.

Limitations and Future Work

  • Limitations:
    • The study acknowledges variability in performance across different medical questions, suggesting that model performance is highly dependent on question complexity, medical specialty, and the specific medical knowledge required.
  • GPT-5-mini's suboptimal performance might be due to issues in the evaluation setup, model configuration, or the specific implementation used.
  • The paper notes that while zero-shot LLMs are strong in general language understanding, they are not fine-tuned for medical contexts, which can affect exact lexical matching.
  • Future Work:
    • The paper aims to serve as a standardized setting for future studies to minimize model size, computational resources, and maximize clinical utility in medical NLP applications.
  • The findings highlight the need for future evaluations to incorporate factual correctness, ethical compliance, and explainability, moving beyond traditional automatic metrics.
  • Further research could explore the impact of domain-specific fine-tuning and more advanced LLM-as-a-Judge methodologies.

Summary updated

Abstract

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.
BibTeX
@misc{adib2026assessing,
  author = {Shefayat E. Shams Adib and Ahmed Alfey Sani and Ekramul Alam Esham and Ajwad Abrar and Tareque Mohmud Chowdhury},
  title = {{Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation}},
  year = {2026},
  eprint = {2602.14564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.14564},
}
Figure