Abstract
While we pay attention to the latest advances in clinical natural language processing (NLP), we can notice some resistance in the clinical and translational research community to adopt NLP models due to limited transparency, interpretability, and usability. In this study, we proposed an open natural language processing development framework. We evaluated it through the implementation of NLP algorithms for the National COVID Cohort Collaborative (N3C). Based on the interests in information extraction from COVID-19 related clinical notes, our work includes 1) an open data annotation process using COVID-19 signs and symptoms as the use case, 2) a community-driven ruleset composing platform, and 3) a synthetic text data generation workflow to generate texts for information extraction tasks without involving human subjects. The corpora were derived from texts from three different institutions (Mayo Clinic, University of Kentucky, University of Minnesota). The gold standard annotations were tested with a single institution's (Mayo) ruleset. This resulted in performances of 0.876, 0.706, and 0.694 in F-scores for Mayo, Minnesota, and Kentucky test datasets, respectively. The study as a consortium effort of the N3C NLP subgroup demonstrates the feasibility of creating a federated NLP algorithm development and benchmarking platform to enhance multi-institution clinical NLP study and adoption. Although we use COVID-19 as a use case in this effort, our framework is general enough to be applied to other domains of interest in clinical NLP.
Introduction
Despite the rapid evolution of Natural Language Processing (NLP) technologies, their integration into routine clinical research remains limited. This "adoption gap" is often driven by a lack of transparency and the "it works on my machine" phenomenon—where an NLP model developed at one institution fails to perform when deployed at another. These failures typically stem from two root causes: heterogeneity in Extraction, Transformation, and Load (ETL) processes across different Electronic Health Record (EHR) systems, and variations in how human experts annotate "gold standard" data.
To address these challenges, we present an open, federated framework designed to facilitate collaborative NLP development. By leveraging common data models and privacy-preserving evaluation methods, this framework enables the creation of robust, multi-site clinical NLP tools.
The Open NLP Framework
Our proposed framework is structured into three functional layers—Ingestion, Processing, and Persistence—designed specifically for interoperability and clinical trust.
Standardized Data Integration
To overcome ETL heterogeneity, the framework utilizes the OMOP Common Data Model (CDM). By specifically targeting the NOTE and NOTE_NLP tables, the pipeline becomes "plug-and-play" for any institution already utilizing the OMOP standard. This ensures that data structures are consistent regardless of the underlying EHR vendor.
Interpretable Symbolic NLP
While deep learning dominates modern NLP, our framework utilizes a symbolic (rule-based) approach via MedTagger and the ConText algorithm. This choice prioritizes transparency; clinicians and researchers can inspect, understand, and refine the logic used to identify clinical concepts, which is vital for clinical validation.
Collaborative Development and Privacy
To facilitate multi-site cooperation, we implemented two key components:
- Rule Editor and Dictionary Builder: A web-based interface that allows domain experts from different sites to collaboratively synthesize local knowledge into a unified ruleset.
- NLP Sandbox: A federated evaluation environment using Docker containers. This allows algorithms to be sent to the data, ensuring that Protected Health Information (PHI) never leaves the local institution's firewall. Only aggregated performance metrics are shared.
Case Study: COVID-19 Symptom Extraction
We evaluated the framework using a case study focused on 20 COVID-19 signs and symptoms (e.g., dyspnea, fever, anosmia) identified by the CDC and Mayo Clinic. The study involved clinical notes from three distinct institutions: Mayo Clinic, the University of Kentucky (UKen), and the University of Minnesota (UMN).
The initial corpus consisted of 369 randomly selected and de-identified clinical notes, supplemented by synthetic text generation to ensure robust testing without compromising patient privacy.
Results and Discussion
Our evaluation revealed a significant "performance gap" when models were restricted to single-site data. A ruleset developed exclusively on Mayo Clinic data performed exceptionally well at Mayo (0.876 F1) but saw a sharp decline when applied to UMN (0.706) and UKen (0.694).
However, by utilizing the collaborative framework to refine the rulesets using training data from all three sites, we observed substantial improvements across the board:
| Site | Single-Site Ruleset (F1) | Multi-Site Refined Ruleset (F1) |
|---|---|---|
| Mayo Clinic | 0.876 | 0.884 |
| University of Minnesota | 0.706 | 0.769 |
| University of Kentucky | 0.694 | 0.806 |
The most dramatic improvement occurred at the University of Kentucky, where the F1-score increased by over 11 points.
Challenges in Clinical Consensus
Error analysis highlighted that the primary drivers of false positives were contextual. Symptoms mentioned in the context of patient education (e.g., "Patient advised to monitor for fever") or as side effects of treatment were often misidentified as active symptoms. Furthermore, Inter-annotator agreement (IAA) varied significantly between sites (ranging from 0.686 to 0.211), suggesting that local documentation cultures heavily influence what is recorded as "clinical truth."
Conclusion
This study demonstrates the feasibility of a federated, community-driven approach to clinical NLP. By moving away from mono-institutional silos and toward a standardized, transparent framework, the research community can develop more generalizable tools for EHR-based research. Future efforts will focus on scaling this framework across the broader National COVID Cohort Collaborative (N3C) and automating the local fine-tuning process to further reduce the burden on human annotators.
Summary updated