An Open Natural Language Processing Development Framework for EHR-based Clinical Research: A case demonstration using the National COVID Cohort Collaborative (N3C)

Sijia Liu, Andrew Wen, Liwei Wang, Huan He, Sunyang Fu, Robert F. Miller, Andrew E. Williams, Daniel R. Harris, Ramakanth Kavuluru, Mei Liu, Noor Abu-el-rub, Dalton Schutte, Rui Zhang, Masoud Rouhizadeh, John D. Osborne, Yongqun He, Ümit Topaloĝlu, Stephanie Hong, Joel Saltz, Thomas Schaffter, Emily Pfaff, Christopher G. Chute, Timothy Q. Duong, Melissa Haendel, Rafael López Fuentes, Peter Szolovits, Hua Xu, Hongfang Liu, National Covid Cohort Collaborative, Natural Language Processing, Subgroup

Abstract

While we pay attention to the latest advances in clinical natural language processing (NLP), we can notice some resistance in the clinical and translational research community to adopt NLP models due to limited transparency, interpretability, and usability. In this study, we proposed an open natural language processing development framework. We evaluated it through the implementation of NLP algorithms for the National COVID Cohort Collaborative (N3C). Based on the interests in information extraction from COVID-19 related clinical notes, our work includes 1) an open data annotation process using COVID-19 signs and symptoms as the use case, 2) a community-driven ruleset composing platform, and 3) a synthetic text data generation workflow to generate texts for information extraction tasks without involving human subjects. The corpora were derived from texts from three different institutions (Mayo Clinic, University of Kentucky, University of Minnesota). The gold standard annotations were tested with a single institution's (Mayo) ruleset. This resulted in performances of 0.876, 0.706, and 0.694 in F-scores for Mayo, Minnesota, and Kentucky test datasets, respectively. The study as a consortium effort of the N3C NLP subgroup demonstrates the feasibility of creating a federated NLP algorithm development and benchmarking platform to enhance multi-institution clinical NLP study and adoption. Although we use COVID-19 as a use case in this effort, our framework is general enough to be applied to other domains of interest in clinical NLP.

Introduction

Despite the rapid evolution of Natural Language Processing (NLP) technologies, their integration into routine clinical research remains limited. This "adoption gap" is often driven by a lack of transparency and the "it works on my machine" phenomenon—where an NLP model developed at one institution fails to perform when deployed at another. These failures typically stem from two root causes: heterogeneity in Extraction, Transformation, and Load (ETL) processes across different Electronic Health Record (EHR) systems, and variations in how human experts annotate "gold standard" data.

To address these challenges, we present an open, federated framework designed to facilitate collaborative NLP development. By leveraging common data models and privacy-preserving evaluation methods, this framework enables the creation of robust, multi-site clinical NLP tools.

The Open NLP Framework

Our proposed framework is structured into three functional layers—Ingestion, Processing, and Persistence—designed specifically for interoperability and clinical trust.

Standardized Data Integration

To overcome ETL heterogeneity, the framework utilizes the OMOP Common Data Model (CDM). By specifically targeting the NOTE and NOTE_NLP tables, the pipeline becomes "plug-and-play" for any institution already utilizing the OMOP standard. This ensures that data structures are consistent regardless of the underlying EHR vendor.

Interpretable Symbolic NLP

While deep learning dominates modern NLP, our framework utilizes a symbolic (rule-based) approach via MedTagger and the ConText algorithm. This choice prioritizes transparency; clinicians and researchers can inspect, understand, and refine the logic used to identify clinical concepts, which is vital for clinical validation.

Collaborative Development and Privacy

To facilitate multi-site cooperation, we implemented two key components:

  1. Rule Editor and Dictionary Builder: A web-based interface that allows domain experts from different sites to collaboratively synthesize local knowledge into a unified ruleset.
  2. NLP Sandbox: A federated evaluation environment using Docker containers. This allows algorithms to be sent to the data, ensuring that Protected Health Information (PHI) never leaves the local institution's firewall. Only aggregated performance metrics are shared.

Case Study: COVID-19 Symptom Extraction

We evaluated the framework using a case study focused on 20 COVID-19 signs and symptoms (e.g., dyspnea, fever, anosmia) identified by the CDC and Mayo Clinic. The study involved clinical notes from three distinct institutions: Mayo Clinic, the University of Kentucky (UKen), and the University of Minnesota (UMN).

The initial corpus consisted of 369 randomly selected and de-identified clinical notes, supplemented by synthetic text generation to ensure robust testing without compromising patient privacy.

Results and Discussion

Our evaluation revealed a significant "performance gap" when models were restricted to single-site data. A ruleset developed exclusively on Mayo Clinic data performed exceptionally well at Mayo (0.876 F1) but saw a sharp decline when applied to UMN (0.706) and UKen (0.694).

However, by utilizing the collaborative framework to refine the rulesets using training data from all three sites, we observed substantial improvements across the board:

Site Single-Site Ruleset (F1) Multi-Site Refined Ruleset (F1)
Mayo Clinic 0.876 0.884
University of Minnesota 0.706 0.769
University of Kentucky 0.694 0.806

The most dramatic improvement occurred at the University of Kentucky, where the F1-score increased by over 11 points.

Challenges in Clinical Consensus

Error analysis highlighted that the primary drivers of false positives were contextual. Symptoms mentioned in the context of patient education (e.g., "Patient advised to monitor for fever") or as side effects of treatment were often misidentified as active symptoms. Furthermore, Inter-annotator agreement (IAA) varied significantly between sites (ranging from 0.686 to 0.211), suggesting that local documentation cultures heavily influence what is recorded as "clinical truth."

Conclusion

This study demonstrates the feasibility of a federated, community-driven approach to clinical NLP. By moving away from mono-institutional silos and toward a standardized, transparent framework, the research community can develop more generalizable tools for EHR-based research. Future efforts will focus on scaling this framework across the broader National COVID Cohort Collaborative (N3C) and automating the local fine-tuning process to further reduce the burden on human annotators.

Summary updated

Abstract

While we pay attention to the latest advances in clinical natural language processing (NLP), we can notice some resistance in the clinical and translational research community to adopt NLP models due to limited transparency, interpretability, and usability. In this study, we proposed an open natural language processing development framework. We evaluated it through the implementation of NLP algorithms for the National COVID Cohort Collaborative (N3C). Based on the interests in information extraction from COVID-19 related clinical notes, our work includes 1) an open data annotation process using COVID-19 signs and symptoms as the use case, 2) a community-driven ruleset composing platform, and 3) a synthetic text data generation workflow to generate texts for information extraction tasks without involving human subjects. The corpora were derived from texts from three different institutions (Mayo Clinic, University of Kentucky, University of Minnesota). The gold standard annotations were tested with a single institution's (Mayo) ruleset. This resulted in performances of 0.876, 0.706, and 0.694 in F-scores for Mayo, Minnesota, and Kentucky test datasets, respectively. The study as a consortium effort of the N3C NLP subgroup demonstrates the feasibility of creating a federated NLP algorithm development and benchmarking platform to enhance multi-institution clinical NLP study and adoption. Although we use COVID-19 as a use case in this effort, our framework is general enough to be applied to other domains of interest in clinical NLP.
BibTeX
@article{liu2023open,
  author = {Sijia Liu and Andrew Wen and Liwei Wang and Huan He and Sunyang Fu and Robert F. Miller and Andrew E. Williams and Daniel R. Harris and Ramakanth Kavuluru and Mei Liu and Noor Abu-el-rub and Dalton Schutte and Rui Zhang and Masoud Rouhizadeh and John D. Osborne and Yongqun He and Ümit Topaloĝlu and Stephanie Hong and Joel Saltz and Thomas Schaffter and Emily Pfaff and Christopher G. Chute and Timothy Q. Duong and Melissa Haendel and Rafael López Fuentes and Peter Szolovits and Hua Xu and Hongfang Liu and National COVID Cohort Collaborative and Natural Language Processing and Subgroup and National COVID Cohort Collaborative},
  title = {{An Open Natural Language Processing Development Framework for EHR-based Clinical Research: A case demonstration using the National COVID Cohort Collaborative (N3C)}},
  journal = {J. Am. Medical Informatics Assoc},
  year = {2023},
  doi = {10.1093/jamia/ocad134},
}
Figure