Abstract
We introduce CACER, a novel corpus with fine-grained annotations for medical problems, drugs, and their relationships in clinical narratives of oncology notes. State-of-the-art transformer models achieved performance comparable to IAA for several extraction tasks.
Introduction
Clinical notes are a vital repository of patient history, yet their unstructured nature makes them difficult to analyze at scale. In oncology, narratives describe intricate sequences of drug cycles, evolving symptoms, and treatment side effects. While existing systems can extract basic diagnoses, they often fail to capture the interconnectedness of clinical elements—such as identifying which specific drug caused a particular symptom or which medication improved a condition.
To bridge this gap, we present CACER (Clinical Concept Annotations for Cancer Events and Relations), a high-quality dataset featuring over 48,000 fine-grained annotations. Our research evaluates the efficacy of both specialized BERT-based models and large-scale generative language models (GLMs) in extracting these complex clinical events and their relationships.
Methods
The extraction process was framed as two primary tasks:
- Event Extraction (EE): Identifying core "triggers" (medical problems or drugs) and their associated "arguments" (attributes such as severity, duration, or anatomical site).
- Relation Extraction (RE): Determining the links between these events (e.g., Drug Causes Problem or Drug Administered for Problem).
We benchmarked several architectures:
- BERT-based models: Bio+ClinicalBERT, utilizing specialized frameworks like SpERT and PL-Marker for joint entity and relation extraction.
- Generative Language Models (GLMs): Fine-tuned Flan-T5 and Llama3-8B, alongside GPT-4 evaluated via In-Context Learning (ICL).
- Prompt Engineering: For LLMs, we compared standard generation against a "Question-Answering" (QA) format, where the model selects relationships from a predefined list.
The CACER Corpus
The corpus consists of 575 de-identified clinical notes from the Fred Hutch Cancer Center, focusing on prostate cancer and diffuse large B-Cell lymphoma (DLBCL).
Unlike previous datasets, CACER provides exceptional granularity, including six specific attributes for medical problems: Assertion, Change, Severity, Anatomy, Characteristics, Duration, and Frequency. In total, the dataset contains 48,339 events and 10,449 relations.
Key Results and Discussion
Our benchmarks revealed that for Event Extraction, smaller, domain-specific BERT models performed on par with the much larger Llama3-8B, both achieving an F1 score of approximately 88.0. This performance is remarkably close to the human inter-annotator agreement (IAA) of 88.4, suggesting that models are now as consistent as human experts for this task.
Relation Extraction proved more challenging. The best-performing model, Llama3-QA, reached an F1 of 70.3. Notably, GPT-4 performed poorly (F1 of 61.7) when used via ICL, often struggling with strict formatting or hallucinating attributes. This underscores the importance of fine-tuning for high-stakes clinical applications.
Furthermore, we observed that LLMs excel when tasks are framed as multiple-choice reasoning (QA format) rather than open-ended generation. However, all models struggled with "dense" sentences containing long lists of symptoms, which often led to cascading extraction errors.
Conclusion and Future Work
The CACER corpus demonstrates that specialized models can reach human-level performance in clinical event extraction. However, capturing long-distance relations—linking entities across different sections of a clinical note—remains a significant hurdle.
Future research will focus on:
- Entity Normalization: Mapping extracted terms to standard ontologies like ICD-10.
- Knowledge Integration: Incorporating medical ontologies to improve the distinction between adverse effects and indications.
- Efficiency: Investigating the minimal model scale required to maintain human-level accuracy, thereby reducing the computational overhead of LLMs in healthcare settings.
Summary updated