M3Retrieve: Benchmarking Multimodal Retrieval for Medicine

Arkadeep Acharya, Akash Ghosh, Pradeepika Verma, Kitsuchart Pasupa, Sriparna Saha, Priti Singh

TL;DR

The authors introduce M3Retrieve, the first large-scale multimodal retrieval benchmark tailored specifically for the medical domain. Spanning 16 medical specialties and over 1.2 million documents, it challenges AI models to retrieve clinical information using combined text and image queries. The benchmark reveals a significant performance gap, showing that general-domain retrieval models struggle with the complex, nuanced demands of healthcare data.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) has become a cornerstone for deploying large language models reliably, especially in safety-critical fields like medicine. However, medical data is rarely just text; it inherently combines clinical narratives with visual evidence like X-rays, MRIs, and histopathological slides.

While standard benchmarks like BEIR evaluate text-only retrieval and others like M-BEIR look at general multimodal retrieval, there has been no standardized, large-scale benchmark to evaluate multimodal retrieval in a medical context. Medical retrieval introduces unique hurdles:

  • Complex Terminology: Variable and highly specialized jargon requires systems to map plain-language descriptions to complex clinical terms.
  • Niche Specialties: Retrieval systems must generalize across highly distinct medical fields (e.g., Cardiology vs. Psychiatry).
  • Intricate Image-Text Relationships: Visually similar medical images can signify completely different conditions depending on the patient's accompanying history.

Methods

The authors structured M3Retrieve around four distinct, clinically grounded retrieval tasks to mirror real-world workflows:

  1. Visual Context Retrieval: Using an image and a short caption/query to find the most relevant detailed text passage.
  2. Multimodal Summary Retrieval: Matching a patient's multimodal query (text + image) to an expert-written concise summary.
  3. Query to Image Retrieval: Using a text description or reference image to retrieve a visually and clinically similar image.
  4. Case Study Retrieval: Combining an image and a patient transcript to find the closest matching historical case study.
Figure 1: M3Retreive is a multimodal medical retrieval benchmark comprising samples from four different tasks across multiple healthcare subdomains obtained from a variety of open-sourced data sources resulting in total dataset size of about 800K query-corpus pairs. It encompasses testing of retrieval models across varied retrieval paradigms

To ensure evaluation fairness, the authors collaborated with medical experts to design data-source-specific Relevancy Mappings (assigning graded relevance scores based on how explicitly text segments relate to accompanying figures). They then evaluated 10 baseline models across five paradigms: lexicon-based (BM25), text-based dense encoders (e.g., NV Embed), CLIP-style models, multimodal dense encoders (e.g., MM Embed), and late-interaction models (FLMR).

Datasets

M3Retrieve aggregates 22 manually curated datasets under permissive licenses. The evaluation set comprises 1,238,038 corpus documents and 164,947 multimodal queries. The data is pulled from diverse, high-quality open-access sources:

  • Wikipedia: Extracted via the Wikimedia API for broad medical domain coverage.
  • PubMed & PubMed Central: Sourced via subsets of existing clinical datasets like MultiCaRe and Li et al.'s corpus.
  • Open Access Medical Textbooks: From platforms like OpenStax and Open Oregon State.
  • MedPix 2.0: A comprehensive radiology teaching file provided by the U.S. National Library of Medicine.
  • MMQS Dataset: Derived from HealthcareMagic for healthcare summarization.

Key Insights and Ideas

  • The Multimodal Advantage is Task-Dependent: Multimodal models (like MM Embed) shine when tasks require tight, joint understanding of text and imagery. However, for text-heavy tasks like Summary Retrieval, traditional uni-modal text encoders still dominate. Current multimodal architectures often experience a "modality dilution" where visual processing degrades text-centric performance.
  • The Medical Domain Gap is Real: Top-performing general models scored significantly lower on M3Retrieve than their average scores on general benchmarks like BEIR. For example, state-of-the-art models saw performance drops of 25% to 30% compared to their general-domain baselines, highlighting a critical need for domain-specific pre-training.
  • Lexical Baselines Remain Stubbornly Strong: Traditional BM25 term-matching remains a highly competitive baseline in text-heavy medical disciplines, occasionally outperforming deep learning models due to the exact matching of highly specific medical terms.

Key Results

The models were evaluated using nNDCG@10 as the primary metric.

NDCG@10 scores for the ten baseline retrieval models across the four core tasks.
  • VisualContext Retrieval: The multimodal MM Embed model achieved the highest score (45.47), outperforming the top text-only model, NV Embed (43.28).
  • Summary Retrieval: Text-based dense encoders dominated. NV Embed took the top spot with a score of 89.73, while multimodal models lagged significantly behind.
  • Query to Image Retrieval: The domain-specific MedImageInsight (MII) model led decisively with a score of 43.53, proving the value of contrastive learning on medical image-text pairs.
  • Case Study Retrieval: Performance was universally low across the board, but BM25 held the highest score (11.50), closely followed by NV Embed (10.99), illustrating how difficult it is for neural models to parse long-form clinical histories.

Limitations and Future Work

While M3Retrieve is a vital step forward, the authors note several limitations:

  • Discipline Coverage: It currently covers 16 broad medical domains, which does not encompass all highly specialized medical sub-fields.
  • Document Modality: The benchmark primarily features multimodal queries targeting text-only document stores. Future iterations aim to include fully multimodal corpora (documents containing both text and embedded images).
  • Mapping Verification: The figure-to-paragraph relevancy mappings relied on a localized assumption from the source texts and limited manual review, which may not capture broader contextual relevance perfectly.

Summary updated

Abstract

With the increasing use of RetrievalAugmented Generation (RAG), strong retrieval models have become more important than ever. In healthcare, multimodal retrieval models that combine information from both text and images offer major advantages for many downstream tasks such as question answering, cross-modal retrieval, and multimodal summarization, since medical data often includes both formats. However, there is currently no standard benchmark to evaluate how well these models perform in medical settings. To address this gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark. M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over 1.2 Million text documents and 164K multimodal queries, all collected under approved licenses. We evaluate leading multimodal retrieval models on this benchmark to explore the challenges specific to different medical specialities and to understand their impact on retrieval performance. By releasing M3Retrieve, we aim to enable systematic evaluation, foster model innovation, and accelerate research toward building more capable and reliable multimodal retrieval systems for medical applications. The dataset and the baselines code are available in this github page https://github.com/AkashGhosh/M3Retrieve.
BibTeX
@inproceedings{acharya2025m,
  author = {Arkadeep Acharya and Akash Ghosh and Pradeepika Verma and Kitsuchart Pasupa and Sriparna Saha and Priti Singh},
  title = {{M3Retrieve: Benchmarking Multimodal Retrieval for Medicine}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure