MedEureka: A Medical Domain Benchmark for Multi-Granularity and Multi-Data-Type Embedding-Based Retrieval

Yongqi Fan, Nan Wang, Kui Xue, Jingping Liu, Tong RuanNAACL 2025 · DOI

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone for enhancing the reliability of Large Language Models (LLMs), particularly in high-stakes domains like medicine. The effectiveness of RAG depends heavily on embedding models—algorithms that transform text into mathematical vectors to identify relevant information. However, existing benchmarks like MTEB or BEIR are largely generic and fail to capture the nuances of medical text, which is characterized by dense terminology, structured Electronic Health Records (EHRs), and complex doctor-patient dialogues. MedEureka addresses this gap by providing a specialized, multi-faceted evaluation framework designed specifically for the medical domain.

The MedEureka Framework

MedEureka is a comprehensive benchmark that categorizes medical retrieval tasks across a grid of six data types and four levels of text granularity. This structure allows for a granular assessment of how models handle different formats and lengths of medical information.

  • Data Types: Tables, Literature, Knowledge Bases (KB), Terminology (Term), Electronic Health Records (EHR), and Dialogues.
  • Granularities: Phrase, Sentence, Paragraph, and Document.
The MedEureka benchmark structure, illustrating the relationships between sources, text types, granularities, and specific datasets.

To construct this benchmark, the authors utilized a human-machine collaborative approach. GPT-4o was used to generate initial queries and annotations from raw medical sources, which were then rigorously reviewed and corrected by medical experts and NLP researchers. The resulting benchmark consists of 18 datasets derived from diverse sources, including the Chinese Pharmacopoeia (ChP-1), MSD Manuals, and various open-source patient forums.

Key Insights and Results

The researchers evaluated 15 state-of-the-art models, including proprietary ones like OpenAI’s text-embedding-3-large and open-source models like bge-multilingual-gemma2. The results revealed several critical findings regarding the current state of medical retrieval.

The "Hexagonal Warrior" Myth

Evaluation across the diverse datasets showed that no single embedding model dominates every category. While some models excel in medical dialogue, they may struggle significantly with terminology normalization or structured EHR data. This suggests that "general-purpose" medical models may still need specialized tuning for specific clinical tasks.

Radar charts comparing the performance of various embedding models across different MedEureka datasets, highlighting the variance in model strengths.

Performance Gaps in Professional Knowledge

While most models performed excellently on Tables and Dialogues (often exceeding a 90% Hit Rate), performance plummeted on more complex data types like EHRs and Literature. In these categories, even the best models often failed to exceed 50-60% in Mean Reciprocal Rank (MRR@10). Error analysis indicates that models often fail due to a lack of "professional knowledge" rather than linguistic logic; they may retrieve documents that are semantically similar in wording but medically irrelevant to the specific clinical symptoms in a query.

Granularity and Instructions

Retrieval accuracy is highest when the query and the target share the same granularity (e.g., sentence-to-sentence). Performance drops significantly when models must bridge gaps, such as using a short phrase to find a long document. Furthermore, for models that support instruction-tuning (like E5 or BGE), providing explicit context about the data type and granularity in the prompt significantly boosts retrieval accuracy.

Conclusion and Future Work

MedEureka reveals that while modern embedding models are proficient at general medical conversations, they still face significant hurdles in processing professional clinical records and specialized terminology. Future work will focus on expanding the benchmark to more languages, incorporating multi-level relevance metrics (like nDCG), and developing tasks that require "reasoning-heavy" retrieval across multiple documents.

Summary updated

BibTeX
@inproceedings{fan2025medeureka,
  author = {Yongqi Fan and Nan Wang and Kui Xue and Jingping Liu and Tong Ruan},
  title = {{MedEureka: A Medical Domain Benchmark for Multi-Granularity and Multi-Data-Type Embedding-Based Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.154},
}
Figure