An Analysis Of Protected Health Information Leakage In Deep-Learning Based De-Identification Algorithms

Salman Seyedi, Li Xiong, Shamim Nemati, Gari D. CliffordarXiv 2021 · arXiv

Introduction

To facilitate medical research, clinical data must undergo rigorous de-identification to remove Protected Health Information (PHI) such as names, dates, and locations. Modern Natural Language Processing (NLP) tools, particularly those utilizing Long Short-Term Memory (LSTM) networks, have become the standard for this task. However, the massive parameter space of these deep learning models raises a critical security concern: memorization.

In other domains, large-scale models have been shown to inadvertently "remember" sensitive training data, such as credit card numbers. This study performs an empirical security audit on state-of-the-art de-identification models to determine if sharing a trained model (rather than the raw data) poses a practical re-identification risk.

Methodology

The research focuses on NeuroNER, a prominent open-source tool for named-entity recognition in clinical text. The model architecture consists of three primary layers: token and character-level embeddings (Layer 1), a bi-directional LSTM for label prediction (Layer 2), and a Conditional Random Field (CRF) to ensure structural consistency in output labels (Layer 3).

The three-layer architecture of NeuroNER and the specific points where an adversary might "attach" an attack.

To evaluate potential leakage, the authors employed several adversarial strategies:

  • Statistical Analysis (KS Test): Comparing output probability distributions ($P$) for names present in the training set ("inside") versus novel names ("outside").
  • Membership Inference Attack (MIA): Training "shadow models" to recognize the specific statistical fingerprints that training data leaves on model outputs.
  • Brute-Force Attacks: Attempting to identify training set members by finding optimal probability thresholds.

Experimental Setup

The study utilized the i2b2 Challenge Data, a gold-standard corpus for medical de-identification containing over 1,300 patient records. To simulate "outside" data for adversarial testing, the researchers used extensive external dictionaries containing over 80,000 surnames and approximately 8,000 given names.

Results and Discussion

Statistical Significance vs. Practical Risk

Kolmogorov-Smirnov (KS) tests revealed that the model does behave differently when encountering training data. The resulting p-values were extremely small (often $< 10^{-9}$), confirming that a statistical "memory" of the training data exists. However, the actual overlap in probability distributions is remarkably high, rendering this difference nearly useless for a real-world attacker.

Histogram showing the frequency of output probabilities. Note the heavy overlap between original training data and outside names at the 1.0 mark.

The Role of the CRF Layer

Analysis indicated that the CRF layer (Layer 3) acts as an amplifier for training data patterns. The structural constraints learned by the CRF appear to be a primary source of leakage compared to the LSTM layer alone. Despite this, the NeuroNER model maintained high utility, achieving a precision of 98.43% on the test set.

Attack Feasibility

In controlled settings, the Membership Inference Attack achieved an accuracy of approximately 0.75. While significantly better than random chance, this performance is insufficient for reliable re-identification in real-world, unbalanced scenarios where the "needle in a haystack" problem persists. In brute-force tests, even when an attacker possessed almost complete information about a report, the true name was often buried behind hundreds of other likely candidates.

Conclusion

The findings suggest that while deep-learning de-identification models do exhibit subtle statistical traces of their training data, they do not currently pose a significant risk for PHI leakage when distributed. However, the authors caution that this is not a proof of absolute security. Future work should explore the integration of Differential Privacy (DP) to provide mathematical guarantees against leakage, despite the potential trade-offs in model accuracy. For now, distributing trained models remains a viable path for clinical NLP research, provided practitioners remain vigilant regarding evolving adversarial techniques.

Summary updated

BibTeX
@misc{seyedi2021analysis,
  author = {Salman Seyedi and Li Xiong and Shamim Nemati and Gari D. Clifford},
  title = {{An Analysis Of Protected Health Information Leakage In Deep-Learning Based De-Identification Algorithms}},
  year = {2021},
  eprint = {2101.12099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.12099},
}
Figure