De-identifying Clinical Texts using Biomed-Clinical BERTs and Comprehensive Risk Assessment

Dhivin Shaji, Angel Paul, Lifeng Han, Warren Del-Pinto, Goran Nenadic, Suzan Verberne

Abstract

Clinical notes are essential for medical research but contain sensitive Personally Identifiable Information (PII) protected by regulations such as HIPAA and GDPR. This paper introduces DeIDClinic, an enhanced framework that leverages domain-specific transformer models—BioBERT and ClinicalBERT—alongside rule-based and dictionary-based methods to de-identify clinical text. DeIDClinic provides flexible masking options, including redaction and synthetic replacement, and incorporates a novel risk assessment module to quantify re-identification risks. Our results demonstrate high precision and recall across various PII categories, balancing privacy preservation with research utility.

Introduction

The digitization of healthcare has resulted in vast amounts of clinical data, much of which is stored as unstructured "free-text" in clinical notes. While these notes are invaluable for medical research and the development of clinical AI, they are also a privacy minefield. Unlike structured databases where sensitive columns can be easily dropped, clinical narratives weave names, dates, and locations directly into the medical context.

Strict legal frameworks like the Health Insurance Portability and Accountability Act (HIPAA) and the General Data Protection Regulation (GDPR) mandate the protection of PII. However, overly aggressive de-identification can strip away the context necessary for research, rendering the data useless. The challenge lies in building a system that is highly accurate at identifying PII, flexible in its masking strategies, and capable of quantifying the remaining risk of re-identification.

The DeIDClinic Framework

The authors developed DeIDClinic, an enhanced version of the existing MASK framework. The pipeline is designed to be robust and user-friendly, consisting of four primary stages:

  1. Data Ingestion: A user interface facilitates the bulk upload of clinical letters for processing.
  2. Hybrid Entity Recognition: To maximize coverage, the system employs a "Union of Entities" approach. It combines dictionary lookups for professions and locations, rule-based searches for structured patterns like dates and ages, and fine-tuned transformer models (BioBERT and ClinicalBERT) to capture complex entities.
  3. Flexible Masking Strategies: Users can choose between Redaction (replacing PII with generic placeholders like [NAME]) and Replacement (substituting PII with synthetically generated, realistic data). The latter is particularly valuable for Natural Language Processing (NLP) research, as it preserves the linguistic flow of the text.
  4. Risk Assessment: A novel safety check evaluates the probability of a patient being re-identified after processing, flagging high-risk documents for manual review.
The Enhanced MASK Framework (DeIDClinic) architecture, illustrating the pipeline from ingestion to risk assessment.

Experimental Setup and Datasets

The performance of the DeIDClinic models was evaluated using the i2b2 / UTHealth De-identification and Heart Disease Risk Factors dataset, specifically the 2014 PHI Gold Set 1 and 2. This dataset serves as a standard benchmark for clinical NLP tasks, providing a diverse set of clinical notes with annotated PII.

Results and Analysis

The integration of domain-specific models proved highly effective. BioBERT (pre-trained on PubMed) and ClinicalBERT (pre-trained on MIMIC-III hospital records) outperformed general-purpose models in clinical contexts.

Performance metrics (Precision, Recall, and F1-score) for BioBERT and ClinicalBERT across various PII categories.

As shown in the results, both models achieved weighted average F1-scores above 0.95. BioBERT demonstrated a slight advantage in identifying Names (0.985 F1) and Dates (0.974 F1), while ClinicalBERT performed marginally better on IDs (0.989 F1) and Professions (0.730 F1). The "Profession" category remained the most challenging for both models, likely due to the high variability in how occupational information is recorded in clinical settings.

Discussion and Conclusion

The DeIDClinic framework demonstrates that a hybrid approach—combining the strengths of machine learning with traditional rules and dictionaries—creates a more robust safety net for patient privacy. By offering synthetic replacement, the framework ensures that de-identified data remains useful for training downstream medical AI models. Furthermore, the inclusion of a quantitative risk assessment metric addresses the reality that no automated system is perfect, providing researchers with a clear measure of data safety.

Future work will focus on detailed error analysis to improve the detection of "Profession" entities and testing the framework on multi-institutional datasets to ensure its generalizability across different hospital systems and documentation styles.

Source: Shaji, D., et al. (2025). De-identifying Clinical Texts using Biomed-Clinical BERTs and Comprehensive Risk Assessment. 2025 IEEE 13th International Conference on Healthcare Informatics (ICHI). Code Availability: https://github.com/angelpaulml17/DeIDClinic

Summary updated

Abstract

This work focuses on the de-identification and risk assessment of clinical text using Transformer-based biomedical models BioBERT and ClinicalBERT. The primary objective is to accurately identify and mask sensitive entities while maintaining data utility for research. We conduct a detailed comparative analysis between BioBERT, ClinicalBERT, and other models, including NER-BERT, NER-CRF, and LSTMs. We integrate BioBERT and ClinicalBERT into the MASK framework to facilitate effective de-identification (MASK-BioBERT / ClinicalBERT). We introduce a risk assessment tool that evaluates the potential re-identification risks of clinical data post-de-identification. The developed methods contribute to safeguarding patient privacy while supporting the usability of healthcare data for medical research. Our models and platform are publicly available at https://github.com/angelpaulml17/DeIDClinic
BibTeX
@inproceedings{shaji2025de,
  author = {Dhivin Shaji and Angel Paul and Lifeng Han and Warren Del-Pinto and Goran Nenadic and Suzan Verberne},
  title = {{De-identifying Clinical Texts using Biomed-Clinical BERTs and Comprehensive Risk Assessment}},
  booktitle = {ICHI},
  year = {2025},
  doi = {10.1109/ichi64645.2025.00098},
}
Figure