Analysis of free text in electronic health records for identification of cancer patient trajectories

Kasper Jensen, Cristina Soguero-Ruíz, Karl Øyvind Mikalsen, Rolv‐Ole Lindsetmo, Irene Kouskoumvekaki, Mark Girolami, Stein Olav Skrøvseth, Knut Magne AugestadScientific 2017 · DOI

Abstract

With an aging patient population and increasing complexity in patient disease trajectories, physicians are often met with complex patient histories from which clinical decisions must be made. Due to the increasing rate of adverse events and hospitals facing financial penalties for readmission, there has never been a greater need to enforce evidence-led medical decision-making using available health care data. In the present work, we studied a cohort of 7,741 patients, of whom 4,080 were diagnosed with cancer, surgically treated at a University Hospital in the years 2004-2012. We have developed a methodology that allows disease trajectories of the cancer patients to be estimated from free text in electronic health records (EHRs). By using these disease trajectories, we predict 80% of patient events ahead in time. By control of confounders from 8326 quantified events, we identified 557 events that constitute high subsequent risks (risk > 20%), including six events for cancer and seven events for metastasis. We believe that the presented methodology and findings could be used to improve clinical decision support and personalize trajectories, thereby decreasing adverse events and optimizing cancer treatment.

Introduction

While Electronic Health Records (EHRs) are gold mines of data, the most valuable clinical nuances are often buried in unstructured "free text"—doctor’s notes, nursing reports, and discharge summaries—rather than structured codes like ICD-10. For cancer patients, whose journeys are long and complex, these notes contain the essential "why" and "how" of disease progression. However, free text is notoriously difficult to analyze due to negations, typos, and retrospective mentions of past history.

Researchers have developed a novel pipeline to extract "patient trajectories"—the chronological sequence of health events—directly from this unstructured text. By applying natural language processing (NLP) and causal inference to a cohort of over 7,000 surgical patients, the system achieved an 80% accuracy rate in predicting future clinical events and identified 557 high-risk markers for cancer progression and metastasis.

Methodology

The authors built a multi-stage pipeline to process Norwegian-language EHRs:

  1. NLP & Decompounding: A custom "decompounder" was used to split complex Norwegian words into meaningful parts for matching against the NOR-MeSH vocabulary.
  2. Flexible Matching: The Smith-Waterman algorithm, typically used for genetic sequence alignment, was adapted to allow for flexible matching of medical terms, accounting for linguistic variations.
  3. Contextual Classification: Three Naïve Bayes classifiers categorized extracted terms into real-time events, retrospective history, or administrative noise.
  4. Trajectory Mining: Frequent Item Set (FIS) mining identified sequences of events that repeatedly occur together.
  5. Causal Inference: The team controlled for confounders to distinguish true risk from random correlations, quantifying the probability of moving from one health state to another.

Data Overview

The study utilized a massive dataset from the University Hospital of North Norway (UNN), involving 7,741 patients (4,080 with cancer) who underwent gastrointestinal surgery. The source material included over 1.1 million unstructured EHR documents. As shown in Figure 1, the volume of extracted patient events—categorized into symptoms, medications, and surgical procedures—increased significantly over time, providing a rich temporal dataset for trajectory analysis.

Growth of extracted patient events (symptoms, drugs, and surgeries) from 1999 to 2014.

Key Results and Insights

The model achieved a Positive Predictive Value (PPV) of 80% in predicting upcoming patient events. This high accuracy stems from the discovery that clinical events follow a predictable, non-random "grammar." While randomized trajectories yielded only ~19% predictive value, the model’s ordered trajectories proved far more robust.

From 8,326 quantified markers, the system identified 557 high-risk events (defined as a >20% risk of a subsequent adverse event). Figure 2 highlights the most frequent symptoms and diseases extracted from the free text, which serve as the foundational building blocks for these trajectories.

Frequency of the most common symptoms and diseases extracted from the EHR free text.

Specific precursors for cancer and metastasis were also identified. For instance, dysuria was associated with a 36% risk of cancer, while ischemic disease carried a 26% risk of subsequent metastasis. Crucially, the study emphasized the importance of causal adjustment; for example, the raw risk of cancer following chest pain dropped from 31% to a negligible 4% once confounders were controlled.

Discussion and Future Work

Despite its success, the study faces limitations including selection bias (the cohort was limited to surgical patients) and irregular sampling inherent in EHR data. Moving this framework into real-time clinical settings will require overcoming workflow challenges to prevent alert fatigue among clinicians.

Future work will focus on integrating these open-source frameworks into hospital systems to provide real-time clinical decision support, ultimately personalizing patient trajectories to decrease adverse events and optimize cancer treatment.

Summary updated

Abstract

With an aging patient population and increasing complexity in patient disease trajectories, physicians are often met with complex patient histories from which clinical decisions must be made. Due to the increasing rate of adverse events and hospitals facing financial penalties for readmission, there has never been a greater need to enforce evidence-led medical decision-making using available health care data. In the present work, we studied a cohort of 7,741 patients, of whom 4,080 were diagnosed with cancer, surgically treated at a University Hospital in the years 2004-2012. We have developed a methodology that allows disease trajectories of the cancer patients to be estimated from free text in electronic health records (EHRs). By using these disease trajectories, we predict 80% of patient events ahead in time. By control of confounders from 8326 quantified events, we identified 557 events that constitute high subsequent risks (risk > 20%), including six events for cancer and seven events for metastasis. We believe that the presented methodology and findings could be used to improve clinical decision support and personalize trajectories, thereby decreasing adverse events and optimizing cancer treatment.
BibTeX
@inproceedings{jensen2017analysis,
  author = {Kasper Jensen and Cristina Soguero-Ruíz and Karl Øyvind Mikalsen and Rolv‐Ole Lindsetmo and Irene Kouskoumvekaki and Mark Girolami and Stein Olav Skrøvseth and Knut Magne Augestad},
  title = {{Analysis of free text in electronic health records for identification of cancer patient trajectories}},
  booktitle = {Scientific},
  year = {2017},
  doi = {10.1038/srep46226},
}
Figure