Named Entity Inference Attacks on Clinical LLMs: Exploring Privacy Risks and the Impact of Mitigation Strategies

Adam Sutton, Xi Bai, Kawsar Noor, Thomas Searle, Richard DobsonNAACL 2025

TL;DR

This paper explores the privacy risks of clinical Large Language Models (LLMs) by developing a "name inference attack" to extract sensitive patient names from de-identified text. They demonstrate that even with de-identification, a significant percentage of names can be inferred, highlighting the need for robust privacy measures in healthcare AI.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used in sensitive domains like healthcare for tasks such as de-identification. However, like all machine learning models, LLMs are susceptible to "inference attacks" that can reveal sensitive information they were trained on. This paper focuses on a specific type of inference attack: a "name inference attack" targeting clinical LLMs that have been fine-tuned for de-identification. The core problem is understanding how vulnerable these de-identified clinical texts are to attackers who might try to infer specific sensitive entities, like patient names, and how effective common privacy-preserving techniques are against such attacks.

Methods

The authors propose a "name inference attack" that leverages a decoder-style architecture. The core idea is to feed the de-identified text into the victim LLM (AnonCAT in this case) to obtain its hidden states. These hidden states are then used as input for a separate "attack model" (also a decoder model) trained to predict the sensitive names.

Key methodological aspects include:

  • White-Box vs. Black-Box Attacks: They evaluate the attack's effectiveness in two scenarios: white-box (attacker has full access to the model) and black-box (attacker only has API access to the model's outputs). For black-box attacks, they first employ a "model extraction attack" to create a surrogate model.
  • Top-N Sampling: To increase the chances of successful inference, the attack model generates not just the single most likely name, but the n most likely names. This is a novel approach for attribute inference attacks, aiming to capture potential variations or less confident predictions.
  • Differential Privacy (DP): They investigate the impact of applying Differential Privacy with different epsilon values (0.1, 2, 8) to the AnonCAT model and then attacking these DP-protected models. This allows them to quantify the privacy-utility trade-off.
  • Evaluation Metrics: Two metrics are used:
  • Binary Classification: Checks if the true label (ground truth name) is a sublist of the predicted tokens.
  • Sliding Hamming Distance: Measures the similarity between the predicted name and the ground truth name by comparing overlapping sublists of tokens.

Datasets

  • AnonCAT Training/Fine-tuning: The paper mentions that the AnonCAT de-identification model was fine-tuned on a dataset from University College London Hospital, consisting of 560 documents with 10 critical PII concepts manually annotated. This dataset was independently validated for use in de-identification pipelines.
  • Attack Dataset: For the attribute inference attack, the authors use the 2014 i2b2/UTHealth shared task dataset. This corpus includes 1304 longitudinal medical records describing 296 patients, specifically used for de-identification tasks. For black-box attacks, they also mention using the "n2c2" dataset.

Key Insights and Ideas

  • De-identification is Not Foolproof: Even after de-identification, sensitive entities like names are still vulnerable to inference attacks. The paper demonstrates that a significant percentage of names can be recovered.
  • Top-N Sampling Boosts Attack Success: Generating multiple potential names (Top-N) significantly improves the attack's performance compared to just considering the single most likely prediction. This suggests attackers might not need perfect accuracy but rather a high probability of guessing correctly among a set of options.
  • White-Box is More Powerful: As expected, white-box attacks (with full model access) are more effective than black-box attacks, though the gap in binary classification performance is smaller than in Hamming distance.
  • Differential Privacy Trade-off: Applying Differential Privacy does reduce the effectiveness of the name inference attack, but it comes at the cost of the de-identification model's performance (utility). Lower epsilon values (more privacy) lead to greater performance degradation.
  • Masked Language Models are Resilient (to a degree): The inherent difficulty of inferring sensitive attributes from masked language models means that even with a small privacy budget (epsilon), DP can offer some protection, and the attack performance degradation might be less drastic than anticipated for other model types.

Key Results

  • Name Inference Success: The attack can predict approximately 2% of names from an attack dataset when using only the most likely generated label (Top-1).
  • Top-N Impact: Performance increases as n (the number of top predictions considered) increases.
  • White-Box vs. Black-Box: The white-box attack model outperforms the black-box attack model in inferring names.
  • Differential Privacy Impact:
    • The baseline model (no DP) achieves the highest performance.
  • As epsilon decreases (privacy increases), predictive performance degrades. For instance, epsilon 0.1 shows lower performance metrics (e.g., F1 score of 0.653) compared to no privacy (F1 of 0.976).
  • The paper notes that for epsilon values above 1, the model is considered insufficiently private, while below 1 is safer.

Limitations and Future Work

  • Dataset Distribution Mismatch: The victim models were trained on UK data, while the attack models used US-based datasets (n2c2). Future work could investigate how geographic distribution differences impact attack performance.
  • Focus on Names: The current attack specifically targets names. Future work should explore inferring other sensitive attributes like addresses, ages, and other PII.
  • Transformer Architecture Specificity: The attack model was trained only for transformer architectures. Generalizability to other model types is an open question.
  • Attacker Motivations: The paper raises questions about potential attackers' goals (financial, political, etc.) and how this might influence attack strategies.
  • Beyond Red-Teaming: The authors suggest that red-team inference attacks should be part of a broader security strategy, combined with access controls and monitoring.
  • Generative LLMs: Future work could explore inference attacks on generative LLMs, which are becoming increasingly prevalent.

Summary updated

BibTeX
@inproceedings{sutton2025named,
  author = {Adam Sutton and Xi Bai and Kawsar Noor and Thomas Searle and Richard Dobson},
  title = {{Named Entity Inference Attacks on Clinical LLMs: Exploring Privacy Risks and the Impact of Mitigation Strategies}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure