A Novel System for Extractive Clinical Note Summarization using EHR Data

Jennifer Liang, Ching-Huei Tsou, Ananya PoddarNAACL 2019

TL;DR

This paper introduces a novel system for automatically summarizing clinical notes from Electronic Health Records (EHRs) specifically for managing hypertension and diabetes. By incorporating EHR-specific components beyond standard NLP, their system significantly improves the extraction of relevant information, addressing the critical problem of information overload for physicians.

What Problem Does This Paper Address?

Physicians face a deluge of unstructured clinical notes within EHRs, making it time-consuming and difficult to extract crucial patient information at the point of care. This "information overload" is exacerbated by template use and copy-pasting, leading to note bloat. The authors tackle this by developing a disease-specific extractive summarization system that aims to present only the most important information for managing a patient's hypertension or diabetes mellitus. This is a challenging task due to domain-specific terminology, abbreviations, misspellings, temporal narratives, and the inherent difficulty in distinguishing "important" from "unimportant" information in the context of a specific disease.

Methods

The core of the system is a clinical note processing pipeline that extends basic NLP. It includes:

  • Basic NLP Layer: Tokenization, lemmatization, sentence segmentation, POS tagging, parsing, and medical concept recognition (mapping terms to UMLS concepts). This layer is considered commodity and not the focus of novelty.
  • EHR-Specific Components:
    • Disease Context Identification: Recognizing how a sentence relates to a specific disease, even if the disease isn't explicitly mentioned in that sentence. This is crucial because phrases can have different meanings depending on the disease context.
  • Note Section Classification: Identifying which section of a clinical note a sentence belongs to (e.g., chief complaint, assessment and plan). This leverages common note structures to improve insight extraction.
  • Adverse Drug Event (ADE) Detection: A component trained to identify adverse drug events, which are critical for patient safety.
  • Summarization Approach: The paper focuses on extractive summarization, meaning it selects existing sentences from the note. They model this as a sequence labeling task, considering document-level information, and compare three approaches: a linear SVM, a linear-chain CRF, and a simple CNN.

Datasets

The dataset consists of EHRs from a large ambulatory medical group, specifically focusing on patients with a diagnosis of hypertension and/or diabetes mellitus. Notes authored by physicians or nurse practitioners were selected. Approximately half of the selected notes were manually reviewed to ensure they addressed at least one of the target diseases. The final corpus comprised 3,453 outpatient clinical notes from 762 patients, with an average note length of 138 sentences. The corpus was annotated by 12 internal medicine or family medicine physicians over six months, who were asked to identify information relevant to the management of hypertension or diabetes.

Key Insights and Ideas

  • EHR-Specific Components are Crucial: The paper strongly suggests that generic NLP techniques are insufficient for clinical note summarization. Incorporating features like disease context and note section classification significantly improves performance.
  • Disease Context Matters: A phrase like "sub-optimally controlled" can have different implications depending on whether it refers to diabetes, hypertension, or obesity. Explicitly modeling this context is vital for accurate summarization.
  • Note Structure Aids Understanding: Identifying the section of a note (e.g., Assessment and Plan vs. History of Present Illness) provides valuable global context for understanding the importance of a sentence.
  • Data Scarcity is a Major Hurdle: The authors highlight that limited labeled data due to privacy concerns and annotation costs necessitate a modular approach, where components can be trained on separate, smaller datasets.
  • Subjectivity in Annotation: Even with expert physicians, there's inherent subjectivity in what constitutes an "important" sentence for a summary, leading to inter-annotator agreement scores that, while good, are not perfect.

Key Results

  • The system achieved F-scores of 0.657 for hypertension and 0.679 for diabetes mellitus when all components were included in the pipeline.
  • This represents a significant improvement over a baseline using only unigrams (F-scores of 0.555 and 0.581, respectively).
  • The CRF and CNN models, which consider document-level information, outperformed the linear SVM.
  • An ablation study showed that adding disease context and note section classification features improved performance.

Limitations and Future Work

  • Inter-Annotator Agreement: The authors acknowledge the subjective nature of summarization, leading to an inter-annotator agreement of 0.682-0.697. They suggest a third MD adjudicator for consistent ground truth.
  • Evaluation Metrics: While intrinsic evaluation (precision, recall, F-score) was used, the authors note that these metrics don't fully capture the nuance of clinical importance, especially for rare but critical events like Adverse Drug Events (ADEs). They plan for future qualitative intrinsic and quantitative extrinsic evaluations.
  • Redundancy: Redundant information in notes can lead to situations where the system identifies a sentence with similar content to the ground truth, but not the exact one, negatively impacting scores without reflecting a failure to capture the core information.
  • Future Work: The authors plan to explore more advanced deep learning models and conduct extrinsic evaluations to assess the system's usefulness in real-world clinical practice. They also aim to develop more sophisticated methods for handling temporal narratives and inter-disease relationships.

Summary updated

BibTeX
@inproceedings{liang2019novel,
  author = {Jennifer Liang and Ching-Huei Tsou and Ananya Poddar},
  title = {{A Novel System for Extractive Clinical Note Summarization using EHR Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure