De-identification of clinical notes with pseudo-labeling using regular expression rules and pre-trained BERT

Jiyong An, Jiyun Kim, Leonard Sunwoo, Hyunyoung Baek, Sooyoung Yoo, Seunggeun LeeBmc Medical Informatics Decis. mak 2025 · DOI

TL;DR

This paper tackles the challenge of de-identifying Korean clinical notes, a crucial step for enabling medical research while protecting patient privacy. They introduce a novel semi-supervised approach that combines rule-based regular expressions with a pre-trained Korean BERT model (KoBERT-NER) to achieve high accuracy in identifying and masking Protected Health Information (PHI).

What Problem Does This Paper Address?

The increasing adoption of Electronic Health Records (EHRs) has led to a surge in clinical notes, which are rich sources of information for medical research. However, these notes contain sensitive Protected Health Information (PHI) that must be removed before they can be shared or analyzed. While de-identification methods exist for English clinical texts, there's a significant gap in research for non-English languages, particularly Korean. Existing Korean de-identification methods are limited in scope and reliability, hindering the use of valuable Korean clinical data for research. This paper aims to bridge this gap by developing an effective de-identification strategy for Korean clinical notes.

Methods

The authors propose a two-pronged approach:

  1. Rule-Based De-identification: They developed a comprehensive set of 51 regular expression rules to identify six categories of PHI (DATE, NAME, HOSPITAL, REGION, NUMBER, ETC) within radiology reports. This rule-based system was built using a manually annotated dataset by domain experts.
  2. Semi-Supervised Machine Learning: The regular expressions were then used as "pseudo-labelers" to automatically label a large dataset of clinical notes. This pseudo-labeled data was used to fine-tune a pre-trained Korean BERT model (KoBERT-NER). This semi-supervised approach leverages the strengths of both rule-based systems (for initial labeling) and deep learning models (for contextual understanding and improved accuracy).

Datasets

  • Training and Validation Data: The study utilized a large dataset from Seoul National University Bundang Hospital (SNUBH), comprising 11,181,617 radiology reports and 9,282,477 non-radiology reports.
  • Annotated Data: A subset of 11,182 reports (0.1% of the total) was randomly sampled for annotation. Of these, 1,112 contained PHI and were used to develop the regular expression rules.
  • Test Data: For evaluation, 342 radiology reports and 12 non-radiology reports were manually annotated at the token level.

Key Insights and Ideas

  • The Power of Hybrid Approaches: Combining rule-based methods with machine learning, especially in a semi-supervised manner, significantly boosts de-identification performance, particularly when annotated data is limited.
  • Pseudo-labeling is Effective: Using rule-based systems to generate pseudo-labels for large datasets is a viable strategy to overcome the scarcity of manually annotated data for training deep learning models.
  • Domain-Specific Fine-tuning is Crucial: Off-the-shelf pre-trained models (like base KoBERT-NER) perform poorly on clinical text. Fine-tuning with domain-specific data (even pseudo-labeled) is essential for achieving high accuracy.
  • Context Matters for Names: Identifying names of medical staff and patients requires understanding specific contexts and job titles, which regular expressions can capture to some extent, but BERT's contextual understanding offers a significant advantage.
  • Korean Language Nuances: The study highlights challenges specific to Korean, such as the mix of Korean and English, various date notations, and homonyms that can confuse rule-based systems.

Key Results

  • Rule-Based Approach: Achieved a precision of 97.2%, recall of 93.7%, and F1 score of 96.2% on radiology notes.
  • Semi-Supervised BERT Approach (KoBERT-NER fine-tuned with pseudo-labels): Achieved a precision of 96.5%, recall of 97.6%, and F1 score of 97.1%. This approach notably improved recall, which is critical for de-identification.
  • Impact of Training Data Size: Performance generally increased with the amount of training data used for fine-tuning the BERT model.
  • Non-Radiology Notes Improvement: Adding a small number of non-radiology reports to the training data significantly improved the model's performance on notes from other departments.

Limitations and Future Work

  • Homonym and Typo Sensitivity: The rule-based approach struggles with homonyms and typos, leading to some misclassifications. While manual correction of pseudo-labels helped, a fully automated solution is desired.
  • Generalizability of Rules: The regular expressions were primarily developed for radiology reports from a specific hospital, limiting their direct applicability to other departments or institutions without adaptation.
  • Tokenization Challenges: Both KoBERT and SciBERT showed limitations in properly tokenizing domain-specific medical terminology, suggesting the need for base models pre-trained on more specialized corpora.
  • Limited Test Set Diversity: The evaluation was constrained by the size and diversity of the test set. Future work should involve more comprehensive assessments using larger, external test sets from different institutions to better understand generalizability.
  • Adding More Identifiers: Further improvements could be achieved by adding more identifiers to cover a wider range of PHI and training with an extended set of pseudo-labeled notes.

Summary updated

BibTeX
@article{an2025de,
  author = {Jiyong An and Jiyun Kim and Leonard Sunwoo and Hyunyoung Baek and Sooyoung Yoo and Seunggeun Lee},
  title = {{De-identification of clinical notes with pseudo-labeling using regular expression rules and pre-trained BERT}},
  journal = {Bmc Medical Informatics Decis. mak},
  year = {2025},
  doi = {10.1186/s12911-025-02913-z},
}
Figure