In clinical practice, a medical image is never interpreted in a vacuum. Radiologists constantly compare current scans to prior studies, cross-reference MRI findings with CT scans, and map specific visual abnormalities to detailed report text. However, most AI models for medical "embeddings"—the mathematical representations of this data—are evaluated on narrow, isolated tasks.
A new paper from researchers at the University of Sydney and several partners introduces MedHEB, a massive unified benchmark designed to test whether these models can truly handle the heterogeneous and relational nature of clinical evidence.
TL;DR
MedHEB is a comprehensive benchmark featuring 107 retrieval tasks across 54 datasets that evaluates medical embeddings on their ability to link 2D images, 3D volumes, and text. The authors also introduce MedEmb, a powerful multimodal large language model (MLLM) baseline that significantly outperforms existing models by providing a more balanced and clinically meaningful embedding space.
What Problem Does This Paper Address?
The core challenge is that "clinical similarity" is multifaceted. A model might be excellent at matching a picture of a lung to the word "pneumonia" (global image-text retrieval) but completely fail at matching a 2D X-ray to its corresponding 3D CT scan (modality-bridging) or finding a specific small nodule described in a report (region-level retrieval).
Existing benchmarks are fragmented, focusing on either just text, just 3D images, or simple 2D image-label pairs. This leaves a massive gap: we don't know if a single embedding space can reliably preserve clinical relationships as the modality, appearance, and granularity of the evidence change. MedHEB aims to be the "stress test" for medical foundation models.
Methods
The researchers standardized 107 different tasks under a single ranking protocol: a model encodes a query and a pool of candidates into embeddings, and candidates are ranked by cosine similarity.
To provide a strong baseline, they developed MedEmb (Figure 3). Unlike "dual-tower" models that encode images and text separately, MedEmb uses a single MLLM backbone (based on Qwen3-VL). It serializes text prompts, 2D images, and sampled 3D volumes into a single multimodal token stream.
Key technical innovations in MedEmb include:
- Instruction-Conditioned Contrastive Learning: The model is trained to understand what it is looking for based on a text instruction (e.g., "Find the same category image").
- False-Negative Masking: In medical data, two different "normal" X-rays are clinically equivalent. The authors implemented a masking strategy during training so the model isn't penalized for placing these similar "negatives" close to the query.
- Token-Budgeted Dynamic Batching: This allows the model to efficiently handle everything from short labels to massive 3D volumes in the same training batch.
Datasets
MedHEB is a "benchmark of benchmarks," aggregating 54 source datasets into 107 tasks and 75,233 queries. It spans eight anatomical domains (e.g., Neuro, Chest, Pathology) and three axes of clinical evidence:
- Evidence Form: 2D images, 3D volumes, and clinical text.
- Pairing Type: Image-image, image-text, text-text, and modality-bridging (e.g., MRI T1 to T2).
- Relevance Granularity: Global labels, question-conditioned answers, and localized region-level matching.
Key Insights and Ideas
- The "Generalization" Illusion: Models that perform exceptionally well on standard image-text tasks often "collapse" on modality-bridging or region-level tasks. Strength in one area does not imply a balanced clinical embedding space.
- The 3D Bottleneck: 3D retrieval remains significantly harder than 2D. The gap isn't just about longer inputs; it's about the model's struggle to capture spatial continuity and small, cross-slice findings.
- Compositionality is Hard: As target text becomes more complex (moving from a single label to a full report), performance drops. Models struggle to compress multiple clinical attributes (anatomy, severity, laterality) into a single vector while maintaining precise alignment with visual evidence.
- Instruction Matters: Fine-tuning with explicit instructions (as done with MedEmb) helps the model navigate the embedding space more effectively for specific clinical needs.
Key Results
The results show a stark divide between current public baselines and the proposed MedEmb model.
- Quantitative Superiority: MedEmb-8B achieved an overall Hit@1 of 49.32%, nearly doubling the performance of many established models like SigLIP (14.89%) or BLIP-2 (11.85%).
- Cross-Family Consistency: MedEmb showed much lower "coefficient of variation" across different task types, meaning its performance is more balanced and reliable across the various "stress tests" in the benchmark.
- Human Validation: In a single-blind study, a board-certified clinician preferred the retrieval results of MedEmb-8B in 49.1% of cases, compared to only 23.0% for the strongest public baseline.
Limitations and Future Work
The authors acknowledge several limitations:
- Single-Positive Formulation: Most tasks assume there is only one "correct" match, which simplifies the often-ambiguous reality of clinical relevance.
- Dataset Noise: Using 54 different sources inevitably introduces variations in data quality and labeling conventions.
- Access Restrictions: Due to the sensitive nature of medical data, users must still obtain credentials for certain datasets (like MIMIC-CXR) to fully reconstruct the benchmark using the provided scripts.
Future work will likely focus on improving the "compositional" alignment for long-form reports and further closing the performance gap in 3D volumetric retrieval.
Summary updated