A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients

Edward T. Moseley, Joy T. Wu, Jonathan Welt, John Foote Jr., Patrick D. Tyler, David W. Grant, Eric T. Carlson, Sebastian Gehrmann, Franck Dernoncourt, Leo Anthony Celi

Introduction

Electronic Health Records (EHRs) are a cornerstone of modern clinical research, yet they suffer from a significant "dark data" problem. Structured data, such as ICD billing codes, are primarily optimized for administrative reimbursement and often fail to capture the clinical nuance necessary for predicting patient outcomes. Critical factors like treatment non-adherence or chronic pain are frequently buried within unstructured text notes. To address this, researchers have developed a new expert-annotated dataset of over 2,100 ICU patient notes from the MIMIC-III database, specifically targeting "high-context" medical conditions that are often missing from standard codes.

Methodology

The study focused on "frequent flyers"—patients admitted to the ICU more than three times in a single year—as they are most likely to exhibit complex, high-context conditions. The researchers annotated 1,102 Discharge Summaries and 1,000 Nursing Progress Notes.

The annotation process involved pairs of clinical researchers and resident physicians following rigorous definitions for 13 specific phenotypes. These phenotypes, ranging from Advanced Heart Disease to Substance Abuse, were selected for their relevance to hospital readmission risks and their typical absence from structured billing data.

Table 1: Definitions and examples of the high-context phenotypes annotated in the corpus.

To validate the utility of this corpus, two baseline Natural Language Processing (NLP) models were implemented: a Bag-of-Words (BoW) model with Logistic Regression and a Convolutional Neural Network (CNN).

Key Insights: The Value of Context and Nursing Notes

A primary finding is the "context gap" between billing codes and clinical reality. For instance, a patient might be coded for "Diabetic Ketoacidosis" while the underlying cause—"Treatment Non-Adherence"—is only documented in the text. By including Nursing Notes alongside Discharge Summaries, the researchers captured a more granular, real-time perspective of patient status that is often overlooked in NLP research.

The study also demonstrated high inter-annotator agreement, with many Cohen’s Kappa values exceeding 0.80. This proves that even relatively subjective conditions like depression or non-adherence can be reliably labeled when clinical experts are provided with clear, standardized definitions.

Results: CNN vs. Bag-of-Words

The experimental results underscore the importance of context-aware modeling. The CNN baseline significantly outperformed the simpler BoW model across nearly all categories, particularly where linguistic context is vital for disambiguation.

Table 2: F1-score comparison between Bag-of-Words (BoW) and Convolutional Neural Network (CNN) models across selected phenotypes.

As shown in the results, the CNN achieved a 95% F1-score for Obesity, compared to just 30% for BoW. Similarly, for Depression and Psychiatric Disorders, the CNN reached F1-scores of 84% and 83% respectively, while BoW lagged significantly. However, "hard" phenotypes like Chronic Pain (49% F1 with CNN) remain challenging due to the highly variable vocabulary used to describe them, suggesting that even neural networks require more data or more advanced architectures to generalize in these areas.

Conclusion and Future Work

While the current dataset provides a high-quality benchmark, its reliance on data from a single institution may limit its immediate generalizability. Future research should explore expanding the corpus to other healthcare systems and utilizing more advanced transformer-based architectures to improve detection of the most linguistically complex conditions. This work represents a crucial step toward making the "dark data" of clinical notes actionable for improving patient care and reducing readmissions.

Summary updated

Abstract

A crucial step within secondary analysis of electronic health records (EHRs) is to identify the patient cohort under investigation. While EHRs contain medical billing codes that aim to represent the conditions and treatments patients may have, much of the information is only present in the patient notes. Therefore, it is critical to develop robust algorithms to infer patients' conditions and treatments from their written notes. In this paper, we introduce a dataset for patient phenotyping, a task that is defined as the identification of whether a patient has a given medical condition (also referred to as clinical indication or phenotype) based on their patient note. Nursing Progress Notes and Discharge Summaries from the Intensive Care Unit of a large tertiary care hospital were manually annotated for the presence of several high-context phenotypes relevant to treatment and risk of re-hospitalization. This dataset contains 1102 Discharge Summaries and 1000 Nursing Progress Notes. Each Discharge Summary and Progress Note has been annotated by at least two expert human annotators (one clinical researcher and one resident physician). Annotated phenotypes include treatment non-adherence, chronic pain, advanced/metastatic cancer, as well as 10 other phenotypes. This dataset can be utilized for academic and industrial research in medicine and computer science, particularly within the field of medical natural language processing.
BibTeX
@inproceedings{moseley2020corpus,
  author = {Edward T. Moseley and Joy T. Wu and Jonathan Welt and John Foote Jr. and Patrick D. Tyler and David W. Grant and Eric T. Carlson and Sebastian Gehrmann and Franck Dernoncourt and Leo Anthony Celi},
  title = {{A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients}},
  booktitle = {LREC},
  year = {2020},
  url = {https://openreview.net/forum?id=YpSnNtzO-W},
}
Figure