Hospital Discharge Summarization Data Provenance

Paul Landes, Aaron Chaise, Kunal Patel, Sean Huang, Barbara Di EugenioACL 2023

TL;DR

This paper introduces DSProv, a novel dataset for understanding the provenance of hospital discharge summaries by annotating how information flows from source notes. They developed an unsupervised method to identify these "note matches," aiming to improve automatic summarization by understanding data origins and quality.

What Problem Does This Paper Address?

Generating accurate and reliable hospital discharge summaries is crucial for patient care, but current automatic summarization (AS) methods struggle due to the complexity and redundancy in electronic health records (EHRs). Discharge summaries often heavily rely on copying and pasting text from various source notes (progress notes, consults, test results), leading to potential errors, outdated information, and a lack of true summarization. The authors identify a critical gap: a lack of understanding about where the information in discharge summaries originates and how it's transferred from antecedent notes. This "data provenance" is essential for building robust AS systems that can faithfully represent patient information.

Methods

The core of this work involves two main components:

  • Data Annotation: Three clinical informatics physicians manually annotated 51 hospital admissions from the MIMIC-III corpus. They identified "note matches," which are semantically similar or verbatim text spans in both the discharge summary and its antecedent notes. This involved defining specific criteria for matching text segments.
  • Unsupervised Note Matching: To automate the identification of these note matches, the authors developed a novel unsupervised method. This method leverages the Word Mover algorithm (based on Earth Mover's Distance) to model the flow of information between word embeddings of source and target documents. They then employ Hybrid Semantic Positional Token Clustering to group semantically similar word embeddings and identify potential matches. This approach combines word embeddings with positional information to create more robust matches, aiming to overcome the "Swiss cheese" problem of fragmented matches.

Datasets

  • MIMIC-III Corpus: The primary dataset used is the freely available Medical Information Mart for Intensive Care III (MIMIC-III) Version 1.4. This corpus contains a large collection of de-identified health-related data, including ICU admissions.
  • DSProv Dataset: The authors introduce DSProv, a new dataset consisting of 51 hospital admissions annotated with note matches. This dataset includes detailed information about the source and target text spans, their semantic similarity, and their provenance.

Key Insights and Ideas

  • Provenance is Key: Understanding the origin of information in discharge summaries is a fundamental prerequisite for effective automatic summarization. Simply treating discharge summaries as monolithic text ignores the complex data flow from antecedent notes.
  • High Copy-Paste Rate: The analysis of the DSProv dataset confirms that a significant portion of discharge summaries is copied from antecedent notes, with some sections showing very high overlap.
  • Semantic vs. Surface Similarity: The analysis reveals that while some sections are frequently copied verbatim (high surface similarity), others are paraphrased, leading to low edit distance but high semantic similarity (e.g., high BERTSCORE). This distinction is crucial for AS.
  • Physician Notes Dominate: Physician notes, particularly those related to the "Hospital Course," show a high overlap with discharge summaries, suggesting they are a primary source of information.
  • ECG, Radiology, and Echo are Heavily Copied: Notes from specialties like ECG, Radiology, and Echo exhibit very high copy-paste rates into discharge summaries, likely due to their structured and factual nature.

Key Results

  • The DSProv dataset contains 569 note matches across 291 antecedent notes, covering over 3 million tokens.
  • The analysis shows that "Hospital Course" sections have the highest discharge summary representation (14%), followed by "Labs and Imaging" (14%).
  • "Assessment and Plan" sections from note antecedents show a high overlap (15%) with discharge summaries.
  • The unsupervised method achieved a SemEval partial recall of 69.06 for matching discharge summary spans, indicating good ability to find relevant text. However, precision was lower, suggesting challenges in precisely defining span boundaries.
  • When evaluating note antecedent matching, the SBERT model achieved the highest harmonic mean score of 15.85, outperforming other models and even specialized biomedical models like SapBERT.

Limitations and Future Work

  • MIMIC-III Limitations: The MIMIC-III corpus primarily includes ICU notes. This means that information from patients after transfer to lower-severity departments is missing, limiting the scope of summarization for the entire hospital stay.
  • Unsupervised Model Performance: While the unsupervised method shows promise, its performance (particularly precision) indicates room for improvement.
  • Future Work:
    • Developing supervised methods, potentially using pre-trained biomedical language models fine-tuned for span tagging.
  • Exploring summarization of structured data, such as generating medication history sections from tabular prescribed medicine data.
  • Investigating how to better utilize the provenance information to create more traceable and faithful automatic discharge summaries.

Summary updated

BibTeX
@inproceedings{landes2023hospital,
  author = {Paul Landes and Aaron Chaise and Kunal Patel and Sean Huang and Barbara Di Eugenio},
  title = {{Hospital Discharge Summarization Data Provenance}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure