CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports

Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu

TL;DR

This paper introduces CaseReportBench, a novel, expert-annotated dataset for dense information extraction from clinical case reports, specifically focusing on rare diseases. They also propose and evaluate new prompting and data integration strategies for Large Language Models (LLMs) to tackle this challenging task, finding that category-specific approaches and open-source models often outperform larger, proprietary ones.

What Problem Does This Paper Address?

Diagnosing and managing rare diseases, like Inborn Errors of Metabolism (IEMs), is incredibly challenging. Clinical case reports are a rich source of information for these conditions, but manually extracting detailed, structured data from them is time-consuming, error-prone, and doesn't scale. While LLMs show promise for information extraction, their ability to perform "dense information extraction" – pulling out granular details across multiple predefined categories – in the complex domain of clinical case reports hasn't been systematically evaluated. This paper aims to bridge that gap by creating a benchmark and evaluating LLM performance on this critical task.

Methods

The authors developed CaseReportBench by:

  • Preprocessing: Filtering open-access PMC case reports to focus on relevant sections and exclude irrelevant content (like references or treatment details).
  • Case Report Selection: Using the IEMbase knowledge base to identify case reports related to IEMs.
  • Expert Annotation: Employing a multi-step process with two clinical specialists and a third reviewer to annotate case reports across 14 predefined clinical categories (e.g., Vitals/Hematology, Neurology, Cardiovascular System). They used the Prodigy annotation tool and developed specific metrics (like Token Set Ratio - TSR) to evaluate inter-annotator agreement, especially for dense extractions where exact matches are rare.
  • LLM Evaluation: Testing five LLMs (Qwen2.5 variants, Llama3, GPT-40) using various prompting and data integration strategies. Key strategies include:
  • Filtered Category-Specific Prompting (FCSP): Applying tailored prompts to specific sections of the case report identified by subheadings.
  • Uniform Category-Specific Prompting (UCP): Applying category-specific prompts to the entire text without subheading filtering.
  • Uniform Global Prompting (UGP): A single prompt for all categories across the entire text.
  • Zero-Shot (ZS) and Few-Shot (FS) Prompting: Varying the number of examples provided in the prompt.

Datasets

The primary dataset introduced is CaseReportBench, an expert-annotated benchmark dataset comprising 138 clinical case reports focused on IEMs. This dataset is publicly available. The source data for preprocessing was the PMC Open Access Subset.

Key Insights and Ideas

  • Category-Specific Prompting is King: Tailoring prompts to specific clinical categories (like FCSP) significantly improves LLM performance compared to generic prompts (UGP). This highlights the importance of domain-specific guidance.
  • Open-Source Models Shine: Surprisingly, smaller open-source models like Qwen2.5:7B often outperformed larger, proprietary models like GPT-40 in terms of accuracy and adherence to instructions. This suggests that instruction fidelity and task-specific tuning are more critical than sheer model size for this task.
  • GPT-40's Struggle with Strict Extraction: While GPT-40 is good at general reasoning, it tends to infer, merge details, and deviate from structured formatting, leading to lower performance on strict, verbatim extraction tasks like this.
  • Subheading Filtering Offers Efficiency: FCSP, which uses subheadings to filter text for category-specific prompts, achieved comparable accuracy to UCP but with a notable efficiency gain (reducing computation time).
  • Negative Findings are a Challenge: LLMs still struggle to reliably identify and extract "negative findings" (e.g., "no chest pain"), which are crucial for differential diagnosis.

Key Results

  • Qwen2.5:7B Outperforms: Qwen2.5:7B achieved the highest alignment with expert annotations, followed by Llama3-8B and Qwen2.5:32B. GPT-40 performed the worst.
  • FCSP and UCP Similar Performance: FCSP and UCP showed comparable TSR scores, indicating the effectiveness of category-specific prompting.
  • Efficiency Gains: FCSP reduced computation time by 6.2% compared to UCP.
  • Clinician Validation: A case study using Llama3:8B with FS-FCSP showed that LLM outputs were rated highly for readability, conciseness, and practicality, reducing annotation time by approximately 24 hours for 138 cases. However, limitations in handling complex cases and categorization errors were noted.

Limitations and Future Work

  • Underrepresented Categories: The LYMPH category was underrepresented in the expert annotations, limiting conclusive insights into its extraction performance.
  • Clinician Oversight Needed: Despite high token-level accuracy, LLM outputs still require expert oversight to ensure clinical actionability.
  • Future Directions:
    • Exploring ZSCOT (Zero-Shot Chain-of-Thought) for more models.
  • Developing more sophisticated prompting strategies for complex cases and negative findings.
  • Expanding the dataset to include more diverse rare diseases and clinical scenarios.
  • Investigating methods to improve LLM performance on categories with lower inter-annotator agreement.

Summary updated

BibTeX
@inproceedings{zhang2025casereportbench,
  author = {Xiao Yu Cindy Zhang and Carlos R. Ferreira and Francis Rossignol and Raymond T. Ng and Wyeth Wasserman and Jian Zhu},
  title = {{CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports}},
  booktitle = {Chil},
  year = {2025},
}
Figure