Named Clinical Entity Recognition Benchmark

Wadood M. Abdul, Marco Af Pimentel, Muhammad Umar Salman, Tathagata Raha, Clément Christophe, Praveen K. Kanithi, Nasir Hayat, Ronnie Rajan, Shadab KhanarXiv 2024 · DOI · OpenReview

Introduction

Extracting structured information from unstructured clinical notes—a task known as Named Entity Recognition (NER)—is a foundational component of modern healthcare AI. It enables critical functions ranging from automated medical billing to the identification of candidates for clinical trials. However, the development of robust clinical NER systems has historically been hindered by inconsistent terminology across datasets, varying evaluation metrics, and a lack of comparative data between traditional specialized models and modern Large Language Models (LLMs).

This paper introduces the Clinical NER Leaderboard, a standardized benchmark designed to provide a transparent, unified framework for evaluating both encoder-based and decoder-based architectures in the clinical domain.

The Challenge of Clinical NER

The field currently faces three primary hurdles:

  1. Terminology Chaos: Disparate datasets often use conflicting labels for identical concepts (e.g., "Condition" vs. "Disease").
  2. Evaluation Inconsistency: The absence of a standard metric makes it difficult to compare model performance across different research papers.
  3. The LLM Gap: Most existing benchmarks predate the rise of generative AI, leaving the relative performance of models like GPT-4o or Llama-3 against specialized medical models largely unquantified.

Standardizing the Clinical Taxonomy

To address terminology fragmentation, the researchers performed a "semantic alignment," mapping entity types from multiple sources to the OMOP (Observational Medical Outcomes Partnership) Common Data Model. This standardization ensures that models are tested against consistent definitions across different clinical contexts.

Standardized Entity Types and Examples

The benchmark curates four major open-source clinical datasets re-mapped to this OMOP standard: NCBI Disease (PubMed abstracts), CHIA (clinical trial eligibility), BC5CDR (chemicals and diseases), and BIORED (complex biological relations).

Evaluation Methodology

The framework evaluates three distinct architectural approaches:

  • Encoders: Traditional token-classification models like BioBERT.
  • Decoders: Generative LLMs (e.g., GPT-4o) prompted for structured output.
  • GLiNER Encoders: A hybrid architecture using bi-directional transformers to match text spans against entity embeddings.

Performance is measured using both token-based and span-based metrics. Span-based evaluation is particularly critical for clinical accuracy, as it requires the model to correctly identify the boundaries of complex, multi-word medical phrases.

Definitions of Span Classification Criteria

Key Results: Specialized Models vs. General LLMs

The leaderboard results as of October 2024 reveal a significant trend: specialized, smaller architectures currently outperform massive general-purpose models in clinical extraction tasks.

  • Top Performer: The gliner-multitask-large-v0.5 model led the benchmark with an average Span-based F1-score of 65.67.
  • LLM Performance: While impressive, gpt-4o achieved a lower score of 53.35, and Meta-Llama-3-70B-Instruct followed at 49.40.
  • Extraction vs. Generation: The data suggests that the auto-regressive nature of decoders (generating text one word at a time) is a disadvantage for NER compared to the holistic, bi-directional view provided by encoder-based models.
  • Fine-tuning Impact: For decoder models, those specifically fine-tuned for NER (such as UniNER) significantly outperformed zero-shot general-purpose models.

Insights and Limitations

A notable finding is the "Condition Bias"; models generally perform better at identifying "Conditions" and "Drugs" than "Measurements" or "Procedures," likely due to more consistent training data for the former categories.

The authors acknowledge a persistent "long tail" problem regarding label imbalance. Clinical datasets are often skewed toward common entities, meaning high aggregate scores can sometimes mask poor performance on rarer, yet critical, clinical entities like specific genetic variants.

Conclusion

The Clinical NER Leaderboard demonstrates that for specialized extractive tasks in healthcare, scale does not always equal superior performance. By providing a standardized framework and mapping to the OMOP CDM, this benchmark paves the way for more interoperable and reliable AI deployment in real-world clinical settings. Future work will focus on incorporating more diverse datasets and developing metrics to better account for entity density and label imbalance.

Summary updated

Abstract

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP.
BibTeX
@misc{abdul2024named,
  author = {Wadood M. Abdul and Marco AF Pimentel and Muhammad Umar Salman and Tathagata Raha and Clément Christophe and Praveen K. Kanithi and Nasir Hayat and Ronnie Rajan and Shadab Khan},
  title = {{Named Clinical Entity Recognition Benchmark}},
  year = {2024},
  eprint = {2410.05046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05046},
}
Figure