MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models

Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, Ying Ding

Introduction

As Large Language Models (LLMs) are increasingly integrated into healthcare for tasks like medical question-answering, their tendency to "hallucinate"—generating plausible-sounding but factually incorrect information—poses a severe risk to patient safety. Existing hallucination benchmarks often focus on general knowledge or summarization, failing to capture the specialized terminology and high-stakes nuances of the medical field. Furthermore, most benchmarks do not distinguish between "easy" hallucinations (obvious errors) and "hard" ones (subtle inaccuracies).

An example of a medical hallucination detection task where an LLM incorrectly identifies a hallucinated answer as correct.

To address this gap, researchers developed MedHallu, a rigorously curated dataset of 10,000 medical question-answer pairs designed to test how well AI can spot its own medical hallucinations. MedHallu stratifies hallucinations by both type and difficulty, providing a more granular evaluation of model reliability in clinical contexts.

Methodology

The authors developed a multi-stage pipeline to generate high-quality, deceptive medical hallucinations:

  1. Candidate Generation: Using 10,000 samples from PubMedQA, they prompted an LLM (Qwen2.5-14B) to generate incorrect answers based on four specific medical error categories: misinterpretation of questions, incomplete information, mechanism/pathway misattribution, and methodological fabrication.
  2. Grading & Filtering: To ensure the hallucinations were deceptive, they used an ensemble of models (Gemma2, GPT-4o-mini, and Qwen2.5). If all models were fooled into thinking the hallucination was correct, it was labeled "Hard." If only one was fooled, it was labeled "Easy."
  3. Refinement: For hallucinations that failed initial quality checks, the authors used TextGrad, an iterative optimization framework, to refine the text until it became more natural and harder to detect.
  4. Correctness Check: They employed bidirectional entailment (using a RoBERTa-based NLI model) to ensure the hallucinated answer was semantically distinct from the ground truth, preventing "false hallucinations" where the model just used different wording for the same correct idea.
The MedHallu generation pipeline, illustrating the multi-stage process of candidate generation, grading, and refinement.

Key Findings

The study revealed several critical insights into how LLMs process medical information:

  • Semantic Proximity to Truth: "Hard" hallucinations are semantically much closer to the ground truth than "easy" ones. The most dangerous hallucinations are those that incorporate enough factual context to remain plausible while introducing a critical, meaningful deviation.
  • The Difficulty of Omission: Among the four error categories, "Incomplete Information" proved the hardest for models to detect. It is significantly easier for an AI to spot a fabricated fact than to realize that a partially correct answer is missing vital medical context.
  • Knowledge is Power: Providing the model with relevant medical context (the "World Knowledge" setting) dramatically improved detection scores across the board. For general LLMs, this increased the average F1 score from 0.533 to 0.784—a massive +0.251 gain.
  • The "Not Sure" Safety Valve: Introducing a "not sure" option allowed models to decline answering when confidence was low. This significantly improved precision, which is critical in clinical settings where a "false negative" (failing to catch a hallucination) is more dangerous than a "false positive."

Results

The evaluation covered a wide range of models, from general-purpose giants like GPT-4o to medically fine-tuned models like UltraMedical. GPT-4o emerged as the top performer, achieving the highest overall F1 score of 0.877 when provided with external knowledge. However, its performance dropped to 0.625 on "hard" hallucinations when no external knowledge was provided.

Surprisingly, general-purpose LLMs often outperformed medically fine-tuned models in detecting hallucinations, likely due to their superior reasoning capabilities and larger scale. Furthermore, allowing a "not sure" category improved precision and F1 scores by up to 38% relative to baseline binary detection.

Conclusion and Future Work

MedHallu highlights the significant challenges LLMs face in identifying subtle medical inaccuracies. While providing context and allowing for uncertainty can mitigate risks, the gap in performance on "hard" hallucinations remains a concern. Future work should expand the benchmark beyond PubMedQA to a more diverse array of medical corpora and explore advanced reasoning techniques like Chain-of-Thought (CoT) to further improve detection accuracy.

Summary updated

Abstract

Advancements in Large Language Models (LLMs) and their increasing use in medical question-answering necessitate rigorous evaluation of their reliability.A critical challenge lies in hallucination, where models generate plausible yet factually incorrect outputs.In the medical domain, this poses serious risks to patient safety and clinical decision-making.To address this, we introduce MedHallu, one of the first benchmark specifically designed for medical hallucination detection.Med-Hallu comprises 10,000 high-quality questionanswer pairs derived from PubMedQA, with hallucinated answers systematically generated through a controlled pipeline.Our experiments show that state-of-the-art LLMs, including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical, struggle with this binary hallucination detection task, with the best model achieving an F1 score as low as 0.625 for detecting "hard" category hallucinations.Using bidirectional entailment clustering, we show that harder-to-detect hallucinations are semantically closer to ground truth.Through experiments, we also show incorporating domainspecific knowledge and introducing a "not sure" category as one of the answer categories improves the precision and F1 scores by up to 38% relative to baselines.
BibTeX
@misc{pandit2025medhallu,
  author = {Shrey Pandit and Jiawei Xu and Junyuan Hong and Zhangyang Wang and Tianlong Chen and Kaidi Xu and Ying Ding},
  title = {{MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models}},
  year = {2025},
  eprint = {2502.14302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14302},
}
Figure