Introduction
The "ChatGPT revolution" has significantly impacted healthcare, leading to a surge in Large Language Models (LLMs) designed for clinical tasks such as diagnosis, report generation, and patient interaction. However, the high stakes of medical applications necessitate rigorous evaluation. Unlike creative writing, "hallucinations" in a medical context can lead to dangerous clinical outcomes. Medical data is inherently complex—often multi-modal, messy, and subject to strict privacy regulations like HIPAA. This paper addresses the lack of a unified guide by providing a systematic taxonomy of benchmarks for both discriminative and generative medical tasks.
Methodology
The authors conducted a systematic literature review to identify and categorize benchmark datasets. The findings are organized into two primary frameworks:
- By Modality: Categorizing data into Text (EHRs, dialogues, QA), Captioned Imagery (2D/3D images with reports), and specialized formats (Video, Audio, ECG, and Omics).
- By Task Type: Distinguishing between Discriminative tasks (e.g., Named Entity Recognition, Relation Extraction, and Information Retrieval) and Generative tasks (e.g., Medical Report Generation, Synthetic Data Creation, and Clinical Summarization).
Taxonomy of Medical Benchmarks
The survey serves as a directory for hundreds of datasets. Key text-based benchmarks include MIMIC-III and MIMIC-IV for intensive care data, and MedQA for USMLE-style reasoning. In the imaging domain, datasets like CheXpert and MIMIC-CXR provide massive collections of chest X-rays, while ROCO links radiology images with scientific captions. Specialized modalities are also covered, including surgical videos (Cholec80) and genomic data (TCGA).
Key Insights and Trends
The survey highlights several critical shifts in the medical AI landscape:
- Domain-Specificity: There is a clear evolution from general-purpose models (like BERT) to domain-specific models (like BioBERT or Med-PaLM) that better handle complex medical terminology.
- Synthetic Data: To address privacy concerns, researchers are increasingly utilizing LLMs to generate synthetic clinical dialogues (e.g., NoteChat), allowing for robust training without compromising patient confidentiality.
- Multimodal Integration: The frontier of medical AI is moving toward models that can simultaneously process text and imagery. Benchmarks like MedPix and Quilt are essential for training models to "see" and "read" clinical data concurrently.
- The Language Gap: A significant limitation identified is the English and Chinese bias in current benchmarks, which may hinder the equitable global deployment of medical AI.
Challenges and Future Directions
Despite the proliferation of benchmarks, several hurdles remain:
- Annotation Bottleneck: High-quality medical labeling requires expensive expert clinician time, necessitating better automated labeling techniques.
- Omics Integration: Effective "embedding" methods are needed to help LLMs relate genetic sequences to clinical outcomes.
- The Hallucination Problem: Models still generate plausible but incorrect advice, necessitating the development of fact-checking layers grounded in verified medical knowledge.
- Diversity: There is an urgent need for datasets representing more diverse geographic regions and languages to ensure global AI equity in healthcare.
Summary updated