MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence

Yingshu Li, Shaoyang Zhou, Zhanyu Wang, Yunyi Liu, Xinyu Liang, Xi Zhang, Lingqiao Liu, Lei Wang, Luping ZhouarXiv 2026 · arXiv

In clinical practice, a medical image is never interpreted in a vacuum. Radiologists constantly compare current scans to prior studies, cross-reference MRI findings with CT scans, and map specific visual abnormalities to detailed report text. However, most AI models for medical "embeddings"—the mathematical representations of this data—are evaluated on narrow, isolated tasks.

A new paper from researchers at the University of Sydney and several partners introduces MedHEB, a massive unified benchmark designed to test whether these models can truly handle the heterogeneous and relational nature of clinical evidence.

TL;DR

MedHEB is a comprehensive benchmark featuring 107 retrieval tasks across 54 datasets that evaluates medical embeddings on their ability to link 2D images, 3D volumes, and text. The authors also introduce MedEmb, a powerful multimodal large language model (MLLM) baseline that significantly outperforms existing models by providing a more balanced and clinically meaningful embedding space.

What Problem Does This Paper Address?

The core challenge is that "clinical similarity" is multifaceted. A model might be excellent at matching a picture of a lung to the word "pneumonia" (global image-text retrieval) but completely fail at matching a 2D X-ray to its corresponding 3D CT scan (modality-bridging) or finding a specific small nodule described in a report (region-level retrieval).

Existing benchmarks are fragmented, focusing on either just text, just 3D images, or simple 2D image-label pairs. This leaves a massive gap: we don't know if a single embedding space can reliably preserve clinical relationships as the modality, appearance, and granularity of the evidence change. MedHEB aims to be the "stress test" for medical foundation models.

Methods

The researchers standardized 107 different tasks under a single ranking protocol: a model encodes a query and a pool of candidates into embeddings, and candidates are ranked by cosine similarity.

To provide a strong baseline, they developed MedEmb (Figure 3). Unlike "dual-tower" models that encode images and text separately, MedEmb uses a single MLLM backbone (based on Qwen3-VL). It serializes text prompts, 2D images, and sampled 3D volumes into a single multimodal token stream.

The MedEmb architecture uses a single MLLM to process interleaved text and medical image inputs into a unified embedding.

Key technical innovations in MedEmb include:

  • Instruction-Conditioned Contrastive Learning: The model is trained to understand what it is looking for based on a text instruction (e.g., "Find the same category image").
  • False-Negative Masking: In medical data, two different "normal" X-rays are clinically equivalent. The authors implemented a masking strategy during training so the model isn't penalized for placing these similar "negatives" close to the query.
  • Token-Budgeted Dynamic Batching: This allows the model to efficiently handle everything from short labels to massive 3D volumes in the same training batch.

Datasets

MedHEB is a "benchmark of benchmarks," aggregating 54 source datasets into 107 tasks and 75,233 queries. It spans eight anatomical domains (e.g., Neuro, Chest, Pathology) and three axes of clinical evidence:

  1. Evidence Form: 2D images, 3D volumes, and clinical text.
  2. Pairing Type: Image-image, image-text, text-text, and modality-bridging (e.g., MRI T1 to T2).
  3. Relevance Granularity: Global labels, question-conditioned answers, and localized region-level matching.
Figure 1: MedHEB design across clinical evidence axes and anatomical domains. The schematic organizes 54 datasets and 107 tasks by evidence form, pairing type, and relevance granularity.

Key Insights and Ideas

  • The "Generalization" Illusion: Models that perform exceptionally well on standard image-text tasks often "collapse" on modality-bridging or region-level tasks. Strength in one area does not imply a balanced clinical embedding space.
  • The 3D Bottleneck: 3D retrieval remains significantly harder than 2D. The gap isn't just about longer inputs; it's about the model's struggle to capture spatial continuity and small, cross-slice findings.
  • Compositionality is Hard: As target text becomes more complex (moving from a single label to a full report), performance drops. Models struggle to compress multiple clinical attributes (anatomy, severity, laterality) into a single vector while maintaining precise alignment with visual evidence.
  • Instruction Matters: Fine-tuning with explicit instructions (as done with MedEmb) helps the model navigate the embedding space more effectively for specific clinical needs.

Key Results

The results show a stark divide between current public baselines and the proposed MedEmb model.

  • Quantitative Superiority: MedEmb-8B achieved an overall Hit@1 of 49.32%, nearly doubling the performance of many established models like SigLIP (14.89%) or BLIP-2 (11.85%).
  • Cross-Family Consistency: MedEmb showed much lower "coefficient of variation" across different task types, meaning its performance is more balanced and reliable across the various "stress tests" in the benchmark.
  • Human Validation: In a single-blind study, a board-certified clinician preferred the retrieval results of MedEmb-8B in 49.1% of cases, compared to only 23.0% for the strongest public baseline.
Figure 4: Benchmark difficulty and cross-relation evaluation. A: Tasks are grouped by public-baseline mean Hit@1 into Collapse, Weak, Medium, and Tractable regimes. B: Paired model scores compare established families (I-I, I-Lab, I-Rep, T-I, T-T) with underexplored relation families (MB-VR, VG).

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Positive Formulation: Most tasks assume there is only one "correct" match, which simplifies the often-ambiguous reality of clinical relevance.
  • Dataset Noise: Using 54 different sources inevitably introduces variations in data quality and labeling conventions.
  • Access Restrictions: Due to the sensitive nature of medical data, users must still obtain credentials for certain datasets (like MIMIC-CXR) to fully reconstruct the benchmark using the provided scripts.

Future work will likely focus on improving the "compositional" alignment for long-form reports and further closing the performance gap in 3D volumetric retrieval.

Summary updated

Abstract

Clinical evidence is inherently relational: medical images are interpreted with prior studies, cross-modal examinations, reports, and localized findings, and retrieving relevant evidence is therefore central to clinical interpretation. Recent foundation models increasingly encode these heterogeneous sources into shared embedding spaces, yet existing embedding evaluation often focuses on isolated matches, leaving unclear whether current models retrieve relevant evidence across changes in modality, appearance, and granularity. We introduce MedHEB, a unified benchmark with 107 retrieval tasks from 54 datasets, including both in-distribution and held-out-source evaluation, across three axes: evidence form (2D images, 3D volumes, text), pairing type (image--image, image--text, text--text, and modality-bridging visual matching), and relevance granularity (global, question-conditioned, region-level). Under a single ranking protocol, MedHEB varies clinically relevant evidence definitions to test whether embedding proximity remains meaningful across heterogeneous evidence, including modality-bridging visual retrieval and region-level text-to-region retrieval. Across representative models, performance is uneven across retrieval families: models strong on conventional image--text or text--text retrieval are less reliable on modality-bridging and region-level tasks. Current embedding spaces capture some clinical relationships well but do not yet provide balanced retrieval across heterogeneous evidence. We further provide MedEmb, an MLLM-based reference embedding model trained on the MedHEB ID split, as a reproducible baseline for unified 2D--3D--text medical retrieval. We release MedHEB with preprocessing scripts, evaluation code, access manifests, and MedEmb to support research on clinically meaningful medical embeddings.
BibTeX
@misc{li2026medheb,
  author = {Yingshu Li and Shaoyang Zhou and Zhanyu Wang and Yunyi Liu and Xinyu Liang and Xi Zhang and Lingqiao Liu and Lei Wang and Luping Zhou},
  title = {{MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence}},
  year = {2026},
  eprint = {2611.02519},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02519},
}
Figure