Large Language Model Benchmarks in Medical Tasks

Xinyuan Song, Lawrence Kq Yan, Ming Li, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Benji Peng, Ziqian Bi, Pohsun Feng, Keyu Chen, Junyu Liu, Qian NiuarXiv 2024 · DOI · arXiv

Introduction

The "ChatGPT revolution" has significantly impacted healthcare, leading to a surge in Large Language Models (LLMs) designed for clinical tasks such as diagnosis, report generation, and patient interaction. However, the high stakes of medical applications necessitate rigorous evaluation. Unlike creative writing, "hallucinations" in a medical context can lead to dangerous clinical outcomes. Medical data is inherently complex—often multi-modal, messy, and subject to strict privacy regulations like HIPAA. This paper addresses the lack of a unified guide by providing a systematic taxonomy of benchmarks for both discriminative and generative medical tasks.

Methodology

The authors conducted a systematic literature review to identify and categorize benchmark datasets. The findings are organized into two primary frameworks:

  1. By Modality: Categorizing data into Text (EHRs, dialogues, QA), Captioned Imagery (2D/3D images with reports), and specialized formats (Video, Audio, ECG, and Omics).
  2. By Task Type: Distinguishing between Discriminative tasks (e.g., Named Entity Recognition, Relation Extraction, and Information Retrieval) and Generative tasks (e.g., Medical Report Generation, Synthetic Data Creation, and Clinical Summarization).

Taxonomy of Medical Benchmarks

The survey serves as a directory for hundreds of datasets. Key text-based benchmarks include MIMIC-III and MIMIC-IV for intensive care data, and MedQA for USMLE-style reasoning. In the imaging domain, datasets like CheXpert and MIMIC-CXR provide massive collections of chest X-rays, while ROCO links radiology images with scientific captions. Specialized modalities are also covered, including surgical videos (Cholec80) and genomic data (TCGA).

Table I: Taxonomy of text-based medical benchmarks including EHR, medical conversations, and QA datasets.

The survey highlights several critical shifts in the medical AI landscape:

  • Domain-Specificity: There is a clear evolution from general-purpose models (like BERT) to domain-specific models (like BioBERT or Med-PaLM) that better handle complex medical terminology.
  • Synthetic Data: To address privacy concerns, researchers are increasingly utilizing LLMs to generate synthetic clinical dialogues (e.g., NoteChat), allowing for robust training without compromising patient confidentiality.
  • Multimodal Integration: The frontier of medical AI is moving toward models that can simultaneously process text and imagery. Benchmarks like MedPix and Quilt are essential for training models to "see" and "read" clinical data concurrently.
  • The Language Gap: A significant limitation identified is the English and Chinese bias in current benchmarks, which may hinder the equitable global deployment of medical AI.

Challenges and Future Directions

Despite the proliferation of benchmarks, several hurdles remain:

  • Annotation Bottleneck: High-quality medical labeling requires expensive expert clinician time, necessitating better automated labeling techniques.
  • Omics Integration: Effective "embedding" methods are needed to help LLMs relate genetic sequences to clinical outcomes.
  • The Hallucination Problem: Models still generate plausible but incorrect advice, necessitating the development of fact-checking layers grounded in verified medical knowledge.
  • Diversity: There is an urgent need for datasets representing more diverse geographic regions and languages to ensure global AI equity in healthcare.

Summary updated

Abstract

With the increasing application of large language models (LLMs) in the medical domain, evaluating these models' performance using benchmark datasets has become crucial. This paper presents a comprehensive survey of various benchmark datasets employed in medical LLM tasks. These datasets span multiple modalities including text, image, and multimodal benchmarks, focusing on different aspects of medical knowledge such as electronic health records (EHRs), doctor-patient dialogues, medical question-answering, and medical image captioning. The survey categorizes the datasets by modality, discussing their significance, data structure, and impact on the development of LLMs for clinical tasks such as diagnosis, report generation, and predictive decision support. Key benchmarks include MIMIC-III, MIMIC-IV, BioASQ, PubMedQA, and CheXpert, which have facilitated advancements in tasks like medical report generation, clinical summarization, and synthetic data generation. The paper summarizes the challenges and opportunities in leveraging these benchmarks for advancing multimodal medical intelligence, emphasizing the need for datasets with a greater degree of language diversity, structured omics data, and innovative approaches to synthesis. This work also provides a foundation for future research in the application of LLMs in medicine, contributing to the evolving field of medical artificial intelligence.
BibTeX
@misc{song2024large,
  author = {Xinyuan Song and Lawrence KQ Yan and Ming Li and Yichao Zhang and Caitlyn Heqi Yin and Cheng Fei and Benji Peng and Ziqian Bi and Pohsun Feng and Keyu Chen and Junyu Liu and Qian Niu},
  title = {{Large Language Model Benchmarks in Medical Tasks}},
  year = {2024},
  eprint = {2410.21348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21348},
}
Figure