CARE: Extracting Experimental Findings From Clinical Literature

Aakanksha Naik, Bailey Kuehl, Erin Bransom, Doug Downey, Tom Hope

Introduction

While Large Language Models (LLMs) have demonstrated remarkable capabilities in summarizing general text, precise Information Extraction (IE) from scientific literature remains a significant bottleneck. In the medical domain, existing IE datasets often rely on the PICO framework (Population, Intervention, Comparator, Outcome). However, this framework is frequently too reductive to capture the nuanced reality of clinical findings.

In practice, a clinical result is rarely as simple as "Drug A improves Condition B." Instead, it involves complex, multi-faceted data: "12 patients with coronary issues given 480 mg/day of oral verapamil saw a reduction in attacks from 123 to 31 (p < 0.006)." To address this complexity, researchers have introduced CARE (Clinical Aggregation-oriented Result Extraction), a new dataset and schema designed to capture fine-grained, n-ary relations including specific dosages, numeric p-values, and population subgroups.

The CARE Schema: Capturing Clinical Nuance

The CARE framework moves beyond simple binary relations to model clinical findings as n-ary relations. This approach is designed to handle the structural complexities inherent in scientific writing:

  • N-ary Relations: Rather than identifying simple pairs, the schema allows for complex groupings of multiple entities and their associated attributes.
  • Structural Complexity: The system explicitly addresses "discontinuous spans"—where a single entity is split across different parts of a sentence—and "nested relations," where one finding serves as an attribute for another.
  • Attribute Linking: The schema distinguishes between core Entities (e.g., Population, Treatment) and their specific Attributes (e.g., Age, Dosage, Route), creating a hierarchical structure that preserves the context of the data.

Dataset Composition

The primary contribution of this work is the CARE dataset, which comprises:

  • 700 Annotated Abstracts: These are split evenly between clinical trials (sampled from the EBM-NLP corpus) and case reports (from PubMed).
  • Rich Annotations: The dataset includes over 16,000 entities and 11,000 relations, providing a dense training and evaluation set for complex IE tasks.
  • Cross-Domain Generalization: To demonstrate the versatility of the schema, the authors conducted pilot annotations in Computer Science and Materials Science. These tests suggested that the underlying "backbone" of the CARE schema generalizes well to other experimental fields.

Benchmarking and Model Performance

The researchers evaluated two distinct categories of models against the CARE dataset: Extractive Baselines (such as PURE and OneIE), which identify text spans directly, and Generative Baselines (such as FLAN-T5 and GPT-4) using zero-shot and in-context learning.

The "Nuance Gap"

The results highlighted a significant performance gap between specialized models and general-purpose LLMs:

  • Entities and Attributes: Specialized extractive models like PURE outperformed LLMs, achieving an F1 score of 55.94 for entities and 61.04 for attributes.
  • Relation Extraction: GPT-4 (5-shot) showed a stronger grasp of complex relationships when provided with sufficient examples, leading the category with an F1 of 55.04.
  • Zero-Shot Limitations: In zero-shot settings, GPT-4’s performance on entities was notably low (F1 of 26.89), as it struggled to capture the exact spans required by the strict scientific schema.

These findings suggest that while LLMs understand relationships, they often fail to identify the precise, granular data points required for automated meta-analyses or personalized medicine applications.

Discussion and Future Directions

The CARE dataset reveals that extracting clinical findings is significantly harder than it looks. Even when provided with "gold" (perfect) entities, GPT-4 only achieved an F1 score of approximately 55 on relation extraction, indicating that the structural complexity of scientific language remains a major hurdle for current AI.

Limitations

  • Annotation Burden: The richness of the CARE schema requires significant human expertise, making it difficult to scale without semi-automated assistance.
  • Language and Evaluation: The current dataset is limited to English. Furthermore, the authors note that "strict" evaluation metrics may unfairly penalize LLMs for minor phrasing variations that are factually correct, suggesting a need for more flexible, yet rigorous, evaluation frameworks.

Conclusion

CARE provides a necessary step toward more sophisticated medical information extraction. By moving away from reductive frameworks and embracing the complexity of n-ary relations, CARE offers a path toward systems that can truly understand and aggregate the messy reality of clinical evidence. Future work will focus on scaling the schema to cover broader experimental findings and exploring its utility in powering automated literature reviews.

Summary updated

BibTeX
@inproceedings{naik2024care,
  author = {Aakanksha Naik and Bailey Kuehl and Erin Bransom and Doug Downey and Tom Hope},
  title = {{CARE: Extracting Experimental Findings From Clinical Literature}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.285},
}
Figure