Robust Benchmarking for Machine Learning of Clinical Entity Extraction

Chloe O'Connell Stanwyck, Ariel Levy, David Sontag, Monica Agrawal

TL;DR

This paper highlights that current state-of-the-art clinical entity extraction systems, while appearing accurate, are brittle and perform poorly on less common concepts. The authors introduce a new annotation framework and demonstrate its utility in creating a more robust benchmark for evaluating these systems, paving the way for more reliable clinical NLP applications.

What Problem Does This Paper Address?

Clinical notes are a treasure trove of patient information, but extracting structured data from this free text is challenging due to the variability of natural language. Clinical Entity Extraction (CEE) aims to solve this by identifying and normalizing concepts (like diseases, symptoms, treatments) to standardized medical vocabularies (like UMLS). While existing systems achieve high overall accuracy, this paper argues that these metrics are misleading. The core problem is that current evaluation methods and datasets don't adequately capture the performance of these systems on less frequent or more complex clinical concepts, leading to a false sense of security about their reliability in real-world healthcare applications. This brittleness can hinder downstream tasks like clinical decision support and research.

Methods

The authors first analyze the performance of top systems from the 2019 n2c2 Shared Task on the Medical Concept Normalization (MCN) Corpus. They then propose a novel annotation framework designed to address the limitations of existing corpora and evaluation methods. Key changes in their framework include:

  • Tagging Multiple CUIs: Allowing multiple Concept Unique Identifiers (CUIs) for a single span if they are equally valid, addressing ambiguity and redundancy in vocabularies.
  • Tagging Subconcepts: Annotating sub-spans of concepts that have their own medical meaning, to better handle inconsistencies in how compound concepts are represented in medical vocabularies.
  • Expanding Concept Coverage: Including a wider range of clinical concepts beyond just problems, tests, and treatments, such as normal findings, anatomical terms, and abbreviations, to better reflect the richness of clinical narratives.

They also developed a custom, open-source annotation tool to facilitate this new framework.

Datasets

  • MCN Corpus: Used for initial analysis of existing systems. This corpus was derived from the 2010 i2b2/VA challenge and contains normalized entity spans from discharge summaries.
  • MIMIC-III Critical Care Database: Used to create a new, small-scale annotated dataset under the proposed framework. This dataset is intended to be open-sourced.

Key Insights and Ideas

  • "High Accuracy" is Misleading: Overall accuracy metrics for clinical entity extraction systems can mask significant performance degradation on less common or more complex concepts.
  • Vocabulary Inconsistencies are a Major Hurdle: The inherent inconsistencies and redundancies within medical vocabularies like UMLS are a primary reason for system brittleness.
  • Subspan Tagging is Crucial: The ability to tag sub-spans of concepts is essential for robust evaluation, as it allows for more accurate assessment of systems when compound concepts are inconsistently represented.
  • Broader Concept Coverage is Necessary: Limiting annotations to only problems, tests, and treatments misses critical information like normal findings and anatomical terms, which are vital for understanding patient context.
  • End-to-End Evaluation is Paramount: In real-world pipelines, entity recognition and normalization are not separate steps. Evaluation frameworks must reflect this to provide a true picture of system performance.

Key Results

  • Performance Drop on Subsets: While overall normalization accuracy on the MCN corpus was high (e.g., Max Acc 85.2% for all spans), performance dropped significantly for "Unseen Text" (69.5%), "Unseen CUI" (69.3%), and especially "Unpopular CUI" (7.0%).
  • Annotator Agreement: The new annotation framework achieved a Jaccard similarity of 0.73 for span selection and 82.6% CUI agreement between two annotators (pre-adjudication), indicating good consistency.
  • MetaMap and cTAKES Performance: These widely used systems showed lower performance compared to top n2c2 systems, with MetaMap achieving 59.0% accuracy and cTAKES 47.1% on the test set. Their performance also varied greatly across different Semantic Types.
  • Impact of Subspan Tagging: The authors' framework identified that cTAKES missed many subspans of compound concepts, which would have been incorrectly flagged as errors under traditional schemas.

Limitations and Future Work

  • Increased Annotation Effort: The proposed framework, by tagging more spans and subspans, increases the annotation workload. However, the authors suggest that a smart annotation tool with suggestions and auto-tagging can mitigate this.
  • Contextual Nuances: The authors acknowledge that minute differences between concepts (e.g., "malabsorption" vs. "malabsorption syndrome") can be difficult for both annotators and systems to distinguish, especially when note context is limited.
  • Critical Care Focus: The current annotation guidelines were developed and evaluated using critical care notes. Future work should evaluate the schema in other clinical settings and note types.
  • Larger Datasets Needed: The paper emphasizes the critical need for substantially larger labeled training sets to drive progress in clinical NLP.
  • Open-Sourcing: The authors plan to open-source their annotation tool and the generated dataset to encourage further research.

Summary updated

Abstract

Clinical studies often require understanding elements of a patient’s narrative that exist only in free text clinical notes. To transform notes into structured data for downstream use, these elements are commonly extracted and normalized to medical vocabularies. In this work, we audit the performance of and indicate areas of improvement for state-of-theart systems. We find that high task accuracies for clinical entity normalization systems on the 2019 n2c2 Shared Task are misleading, and underlying performance is still brittle. Normalization accuracy is high for common concepts (95.3%), but much lower for concepts unseen in training data (69.3%). We demonstrate that current approaches are hindered in part by inconsistencies in medical vocabularies, limitations of existing labeling schemas, and narrow evaluation techniques. We reformulate the annotation framework for clinical entity extraction to factor in these issues to allow for robust end-to-end system benchmarking. We evaluate concordance of annotations from our new framework between two annotators and achieve a Jaccard similarity of 0.73 for entity recognition and an agreement of 0.83 for entity normalization. We propose a path forward to address the demonstrated need for the creation of a reference standard to spur method development in entity recognition and normalization.
BibTeX
@inproceedings{stanwyck2020robust,
  author = {Chloe O'Connell Stanwyck and Ariel Levy and David Sontag and Monica Agrawal},
  title = {{Robust Benchmarking for Machine Learning of Clinical Entity Extraction}},
  booktitle = {MLHC},
  year = {2020},
  url = {https://openreview.net/forum?id=RKi45wQyaS3a},
}
Figure