De-identification of patient notes with recurrent neural networks

Franck Dernoncourt, Ji Young Lee, Özlem Uzuner, Peter Szolovits

TL;DR

This paper introduces the first de-identification system for medical notes based on Artificial Neural Networks (ANNs), specifically using Recurrent Neural Networks (RNNs). By leveraging Long Short-Term Memory (LSTM) architectures, the researchers achieved state-of-the-art performance on two major datasets, effectively eliminating the need for the labor-intensive "feature engineering" required by traditional rule-based or statistical models.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a goldmine for medical research, but they are often locked away to protect patient privacy. In the U.S., HIPAA mandates the removal of 18 types of Protected Health Information (PHI)—such as names, dates, and phone numbers—before notes can be shared.

The challenge is that manual de-identification is prohibitively expensive and slow; for the MIMIC dataset alone, it would cost an estimated $500,000 in human labor. Existing automated systems usually fall into two camps:

  1. Rule-based systems: Brittle, hard to maintain, and easily confused by typos or context (e.g., "Mr. Parkinson" is PHI, but "Parkinson’s disease" is not).
  2. Traditional Machine Learning (e.g., CRFs): Effective, but require "handcrafted features"—manually designed rules that tell the computer what to look for (like "does this word start with a capital letter?").

The authors sought a system that could learn these patterns automatically from the data itself.

Methods

The authors proposed a three-layer ANN model based on Bidirectional LSTMs:

  1. Character-enhanced token embedding layer: This is the "secret sauce." Instead of just looking at whole words, the model looks at individual characters. This allows it to handle misspellings (like "teh" for "the") and recognize sub-word patterns (like prefixes or suffixes) that indicate PHI.
  2. Label prediction layer: A Bi-LSTM that processes the sequence of tokens in both directions (forward and backward) to understand the context of every word.
  3. Label sequence optimization layer: This layer ensures the output makes sense as a sequence. For example, it learns that a "STATE" tag is more likely to be followed by a "ZIP" tag than a "NAME" tag.

They compared this ANN against a high-performing Conditional Random Field (CRF) baseline that they developed using standard morphological and gazetteer (dictionary-based) features.

Datasets

The study utilized two primary benchmarks:

  • i2b2 2014: The largest publicly available de-identification dataset at the time, containing 1,304 notes.
  • MIMIC de-identification dataset: A new dataset assembled by the authors for this study. It is twice as large as the i2b2 dataset, containing 1,635 discharge summaries and over 60,000 PHI instances.

Key Insights and Ideas

  • Semantic Generalization: One of the most striking findings was in the "PROFESSION" category. The ANN significantly outperformed the CRF because word embeddings naturally cluster similar professions (like "glazier," "supervisor," and "CPA") together in vector space. The CRF, by contrast, struggled unless those specific words were already in its manually curated dictionaries.
  • Robustness to Noise: Because the model uses character-level embeddings, it is remarkably resilient to the typos and non-standard tokenization (e.g., "Results02/20/2087") common in clinical notes.
  • Context is King: The Bi-LSTM architecture allows the model to "see" the entire sentence before making a decision, helping it distinguish between a person named "May" and the month of "May."
  • Hybrid Potential: The authors found that combining the outputs of the ANN and the CRF (flagging a word as PHI if either model caught it) further increased recall, suggesting the two models fail in different, complementary ways.

Key Results

The ANN model set new benchmarks for performance:

  • i2b2 2014 Dataset: Achieved an F1-score of 97.85 (97.32% precision, 97.38% recall), outperforming the previous state-of-the-art "Nottingham" system.
  • MIMIC Dataset: Achieved an even higher F1-score of 99.23 (99.06% precision, 99.25% recall).
  • Statistical Significance: The ANN's improvements over the CRF were statistically significant for recall and F1-score on both datasets.

Limitations and Future Work

While the results are impressive, the authors noted a few areas for improvement:

  • The "NAME" Challenge: The F1-score for the "NAME" category—arguably the most sensitive PHI—fell slightly below 98%.
  • Data Scarcity: When training data is extremely limited, the CRF actually performs slightly better because it can rely on its handcrafted rules rather than needing to learn everything from scratch.
  • Future Integration: The authors suggest that adding "gazetteers" (lists of actual patient and staff names from a specific hospital) to the ANN could push the accuracy even closer to 100%.

Summary updated

Abstract

Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of electronic health record databases, the limited number of researchers with access to non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value.
BibTeX
@article{dernoncourt2017de,
  author = {Franck Dernoncourt and Ji Young Lee and Özlem Uzuner and Peter Szolovits},
  title = {{De-identification of patient notes with recurrent neural networks}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2017},
  url = {https://openreview.net/forum?id=6I4tFE4X3L},
}

Loading related content...

Figure