A Weakly-Supervised Named Entity Recognition Machine Learning Approach for Emergency Medical Services Clinical Audit

Han Wang, Wesley Yeung, Qin Xiang Ng, Angeline Tung, Joey Ai Meng Tay, Davin Ryanputra, Marcus Eng Hock Ong, Mengling Feng, Shalini ArulanandamInternational Journal of Environmental Research and Public Health 2021 · DOI

TL;DR

This paper introduces a weakly-supervised machine learning approach for automatically auditing Emergency Medical Services (EMS) clinical reports. By leveraging a small amount of labeled data and a synonym-based pseudo-labeling strategy, they developed a Named Entity Recognition (NER) model that achieves high accuracy in identifying key clinical entities within paramedic reports, significantly reducing the manual effort required for audits.

What Problem Does This Paper Address?

Clinical performance audits are crucial for ensuring quality and continuous improvement in healthcare, particularly in Emergency Medical Services (EMS). These audits typically involve manually reviewing free-text reports written by paramedics to assess adherence to protocols, identify individual performance gaps, and detect systemic issues. However, this manual chart review process is extremely time-consuming, labor-intensive, and resource-heavy, limiting the number of cases that can be audited annually. The paper addresses the need for a more efficient and scalable solution to automate this audit process, thereby enabling more comprehensive quality control and facilitating large-scale data analysis for research.

Methods

The core of this work is a weakly-supervised Named Entity Recognition (NER) approach. Instead of relying on extensive human-labeled data, they employed a strategy to generate "pseudo-labels" for a large corpus of unlabeled text.

  1. Data Preparation: Ambulance incident reports were preprocessed by converting text to lowercase, removing symbols (except "%"), and retaining numbers.
  2. Weakly-Supervised Labeling:
    • They defined 17 clinical entities across three categories (clinical procedure, finding, medication).
  3. A synonym list was created for each entity based on EMS practice.
  4. These synonyms were used to pseudo-label entities in the dataset.
  5. Fuzzy string matching was employed to capture variations and minor spelling mistakes, with a threshold of 1 missing character for strings longer than 5 characters.
  6. A small portion (5%) of the data was human-labeled for verification and fine-tuning.
  7. Model Training: Three models were trained:
  8. BiLSTM-CRF: A Bidirectional Long Short-Term Memory network combined with a Conditional Random Field layer, trained from scratch with a randomly initialized word embedding layer.
  9. BERT-based models: Two variants, BERT-based-uncased and Clinical-BERT, were fine-tuned. These models leverage pre-trained contextualized word embeddings.
  10. Evaluation: Models were evaluated at both the token and entity levels using standard metrics like precision, recall, and F1-score, employing both strict and entity type matching evaluation modes.

Datasets

  • Singapore Civil Defence Force (SCDF) Ambulance Incident Reports: The primary dataset comprised 58,898 unlabeled ambulance incidents recorded between April 1, 2019, and June 30, 2019. After filtering, 44,211 reports were used for the study.
  • Human-labeled data: Only 5% (2,211 reports) of the data were human-labeled for validation and fine-tuning purposes.

Key Insights and Ideas

  • Weak Supervision is Viable for Clinical NLP: The study demonstrates that a weakly-supervised approach, particularly with fuzzy matching and synonym lists, can effectively generate pseudo-labels for a large clinical text corpus, overcoming the bottleneck of manual annotation.
  • BiLSTM-CRF Outperforms BERT in Efficiency: Despite BERT models' strong performance in general NLP tasks, the BiLSTM-CRF model achieved comparable accuracy while being significantly lighter and faster (1-2 orders of magnitude) for this specific EMS clinical audit task. This suggests that for specialized, domain-specific tasks with potentially less complex linguistic structures than general text, simpler architectures can be more practical.
  • Domain-Specific Language Challenges: The paper highlights common NLP challenges in clinical text, such as abbreviations, misspellings, and flexible formatting, which necessitate careful preprocessing and robust model design.
  • Potential for Scalable Audits: The developed model offers a pathway to automate EMS clinical audits, enabling a much higher volume of reviews and freeing up human auditors for more complex tasks or strategic analysis.

Key Results

  • High Performance: The BiLSTM-CRF model achieved an F1 score of 0.981 under entity type matching evaluation and 0.976 under strict evaluation on the test set.
  • Efficiency: The BiLSTM-CRF model had approximately 3.83 million parameters and a checkpoint size of 15 MB, with an inference wall time of 40 ms. In contrast, BERT-based models had over 100 million parameters, checkpoint sizes around 400 MB, and inference times around 270-280 ms.
  • Common Entities: The most frequently observed entities were Electrocardiogram (ECG), Bleeding, and Stroke Assessment.
  • Error Analysis: Key error sources included partial capture of multi-word entities and misspelled words, though the CRF component helped in some cases.

Limitations and Future Work

  • External Validation: The authors acknowledge that the study was conducted on data from a single EMS system (Singapore Civil Defence Force). Further studies are needed to evaluate the external validity of the approach on data from other EMS systems.
  • Spelling Correction: The paper did not explicitly address correcting spelling errors in the text reports, which remains an area for future work.
  • Lighter BERT Models: The study did not experiment with lighter BERT variants (e.g., DistilBERT), which could offer a better trade-off between performance and efficiency.
  • Absent Entity Classes: Some entity classes were absent in the test data, limiting their evaluation.
  • Prospective Evaluation: Future work should involve prospective evaluation of the software in real-world EMS systems to assess its quantitative and qualitative impact on both audits and paramedics.
  • Model Recalibration: Machine learning models for NER may need recalibration over time to adapt to evolving documentation practices and personnel changes.

Summary updated

Abstract

Clinical performance audits are routinely performed in Emergency Medical Services (EMS) to ensure adherence to treatment protocols, to identify individual areas of weakness for remediation, and to discover systemic deficiencies to guide the development of the training syllabus. At present, these audits are performed by manual chart review, which is time-consuming and laborious. In this paper, we report a weakly-supervised machine learning approach to train a named entity recognition model that can be used for automatic EMS clinical audits. The dataset used in this study contained 58,898 unlabeled ambulance incidents encountered by the Singapore Civil Defence Force from 1st April 2019 to 30th June 2019. With only 5% labeled data, we successfully trained three different models to perform the NER task, achieving F1 scores of around 0.981 under entity type matching evaluation and around 0.976 under strict evaluation. The BiLSTM-CRF model was 1~2 orders of magnitude lighter and faster than our BERT-based models. Our proposed proof-of-concept approach may improve the efficiency of clinical audits and can also help with EMS database research. Further external validation of this approach is needed.
BibTeX
@article{wang2021weakly,
  author = {Han Wang and Wesley Yeung and Qin Xiang Ng and Angeline Tung and Joey Ai Meng Tay and Davin Ryanputra and Marcus Eng Hock Ong and Mengling Feng and Shalini Arulanandam},
  title = {{A Weakly-Supervised Named Entity Recognition Machine Learning Approach for Emergency Medical Services Clinical Audit}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2021},
  doi = {10.3390/ijerph18157776},
}
Figure