De-identification of Patient Notes with Recurrent Neural Networks

Franck Dernoncourt, Ji Young Lee, Özlem Uzuner, Peter Szolovits

Abstract

Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes to protect patient confidentiality. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information (PHI) that need to be removed. Manual de-identification is impractical given the size of EHR databases and the frequency of human error. A reliable automated system is therefore of high value. Materials and Methods: We introduce the first de-identification system based on artificial neural networks (ANNs), specifically Recurrent Neural Networks (RNNs), which requires no handcrafted features. We compare its performance on the i2b2 2014 challenge dataset and a larger MIMIC de-identification dataset. Results: Our ANN model outperforms state-of-the-art systems, yielding an F1-score of 97.85 on i2b2 2014 and 99.23 on MIMIC. Conclusion: Findings support the use of ANNs for de-identification, showing superior performance without the need for feature engineering.

Introduction: The Challenge of De-identification

Electronic Health Records (EHRs) are essential for medical research, yet they are often inaccessible due to privacy mandates. In the U.S., HIPAA requires the removal of 18 categories of Protected Health Information (PHI)—including names, dates, and contact details—before notes can be shared.

Manual de-identification is prohibitively expensive; for a dataset like MIMIC, human labor costs are estimated at $500,000. Existing automated solutions typically rely on brittle rule-based systems or traditional machine learning models like Conditional Random Fields (CRFs). While effective, CRFs require extensive "handcrafted features"—manually designed rules that identify morphological patterns (e.g., capitalization). This paper proposes a system that learns these patterns automatically from raw data.

Methodology: A Neural Approach to Sequence Labeling

The researchers developed a three-layer ANN model based on Bidirectional Long Short-Term Memory (Bi-LSTM) architectures. This design allows the model to capture complex linguistic patterns without human intervention.

Architecture of the proposed ANN model, featuring character-enhanced token embeddings and bidirectional context processing.
  1. Character-enhanced token embedding layer: This layer processes individual characters to handle misspellings and recognize sub-word patterns (like prefixes or suffixes) that signal PHI.
  2. Label prediction layer: A Bi-LSTM processes the sequence of tokens in both forward and backward directions, allowing the model to understand the full context of every word.
  3. Label sequence optimization layer: This final layer ensures the output sequence is logically consistent (e.g., ensuring a "STATE" tag is followed by a valid subsequent tag rather than a "NAME" tag).

Datasets and Experimental Setup

The study utilized two primary benchmarks to validate the model:

  • i2b2 2014: A standard de-identification dataset containing 1,304 notes.
  • MIMIC de-identification dataset: A larger dataset assembled for this study, containing 1,635 discharge summaries and over 60,000 PHI instances.

The ANN was compared against a high-performing CRF baseline equipped with standard morphological and dictionary-based features.

Key Results and Performance Analysis

The ANN model established new state-of-the-art benchmarks across both datasets. On the i2b2 2014 dataset, it achieved an F1-score of 97.85, surpassing previous top-tier systems. On the larger MIMIC dataset, performance reached an F1-score of 99.23.

Comparison of F1-scores across PHI categories for the i2b2 and MIMIC datasets, showing the performance of CRF, ANN, and the hybrid CRF+ANN models.

As shown in the performance charts, the ANN consistently outperformed the CRF across almost all PHI categories. Notably, the researchers found that a hybrid approach (CRF+ANN), which flags a word as PHI if either model identifies it, could further increase recall, suggesting that the two models have complementary strengths.

Discussion: Insights and Limitations

Several key insights emerged from the model's performance:

  • Semantic Generalization: The ANN significantly outperformed the CRF in the "PROFESSION" category. Because the model uses word embeddings, it naturally clusters similar terms (e.g., "glazier" and "supervisor"), whereas the CRF struggled with words not present in its manual dictionaries.
  • Robustness to Noise: The character-level embeddings made the model resilient to typos and non-standard tokenization common in clinical settings (e.g., "Results02/20/2087").
  • Contextual Awareness: The Bi-LSTM architecture successfully distinguished between identical words used in different contexts, such as the name "May" versus the month "May."

Despite these successes, challenges remain. The F1-score for the "NAME" category—the most sensitive PHI—remained slightly below 98%. Additionally, when training data is extremely scarce, the CRF's reliance on handcrafted rules can occasionally provide a slight edge over the ANN's data-hungry learning process.

Conclusion

This work demonstrates that Artificial Neural Networks are highly effective for the de-identification of clinical notes. By eliminating the need for labor-intensive feature engineering while simultaneously improving accuracy, this approach paves the way for more efficient and secure sharing of medical data for research purposes.

Summary updated

Abstract

Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information (PHI) that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of EHR databases, the limited number of researchers with access to the non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value. Materials and Methods: We introduce the first de-identification system based on artificial neural networks (ANNs), which requires no handcrafted features or rules, unlike existing systems. We compare the performance of the system with state-of-the-art systems on two datasets: the i2b2 2014 de-identification challenge dataset, which is the largest publicly available de-identification dataset, and the MIMIC de-identification dataset, which we assembled and is twice as large as the i2b2 2014 dataset. Results: Our ANN model outperforms the state-of-the-art systems. It yields an F1-score of 97.85 on the i2b2 2014 dataset, with a recall 97.38 and a precision of 97.32, and an F1-score of 99.23 on the MIMIC de-identification dataset, with a recall 99.25 and a precision of 99.06. Conclusion: Our findings support the use of ANNs for de-identification of patient notes, as they show better performance than previously published systems while requiring no feature engineering.
BibTeX
@article{dernoncourt2017de,
  author = {Franck Dernoncourt and Ji Young Lee and Özlem Uzuner and Peter Szolovits},
  title = {{De-identification of Patient Notes with Recurrent Neural Networks}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2017},
  url = {https://openreview.net/forum?id=6I4tFE4X3L},
}
Figure