Unintended Memorization and Timing Attacks in Named Entity Recognition Models

Rana Salal Ali, Benjamin Zi Hao Zhao, Hassan Jameel Asghar, Tham Nguyen, Ian David Wood, Dali KaafararXiv 2022 · arXiv

TL;DR

This paper demonstrates that Named Entity Recognition (NER) models, specifically the popular spaCy library, are highly vulnerable to privacy attacks that can leak sensitive training data. By exploiting neural memorization and a novel timing side-channel in the tokenizer, researchers achieved up to a 99.23% success rate in identifying whether specific sensitive information—like passwords or medical IDs—was used to train or update a model.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a cornerstone of modern NLP, used to identify names, locations, and sensitive identifiers in text. A common use case is automated redaction: a company might take a pre-trained model and fine-tune it on their private data to better recognize and "black out" sensitive information before sharing documents.

The irony is that the very process of teaching a model to find secrets might cause the model to memorize them. If these models are then offered as a black-box service (e.g., via an API), an adversary could potentially query the model to figure out exactly what was in the private training set. This paper explores whether these "black-box" NER models inadvertently become a liability for the privacy they are meant to protect.

Methods

The researchers investigated two distinct attack vectors against spaCy NER models:

  1. Membership Inference via Memorization: This attack targets the neural network itself. Neural networks tend to be more "confident" (assign higher probability scores) to data they have seen during training. By querying the model with a candidate secret (like a password) and observing the output confidence scores, an attacker can rank candidates to identify the true training sample.
  2. Timing Side-Channel Attack: This novel attack targets the tokenizer rather than the neural weights. As shown in the pipeline below, spaCy maintains an internal vocabulary (a "Strings" table).
Figure 2: The end to end pipeline of an updated NER model in spaCy. Under a black-box setting, the attacker can query the model by passing text to the nlp object, getting labels with their confidence scores.

When the model processes a word it hasn't seen before, it must add that word's features (prefix, suffix, shape, etc.) to its vocabulary. This update takes a measurable amount of extra time. Conversely, words already in the training set (members) are processed faster because they are already in the vocabulary.

Datasets

The study utilized several datasets to simulate real-world scenarios:

  • Public Password Dictionaries: A list of 1 million passwords used to test the memorization of "out-of-distribution" secrets.
  • GUM (Georgetown University Multilayer) Corpus: A dataset of 2,500 annotated documents used to provide a realistic training background.
  • i2b2/UTHealth Corpus: A standard medical dataset containing patient reports with sensitive identifiers like IDs, contact info, and locations.

Key Insights and Ideas

  • Single-Shot Memorization: The researchers found that a model can memorize a secret phrase after just a single insertion into the training set. As shown in the figure below, the "rank" of a secret (its probability relative to others) drops toward zero—meaning it becomes the most probable candidate—in fewer than 10 training epochs.
Figure 3: The extent of model memorization for out-of- distribution words in spaCy (V3.0.3). “Insertions” refer to the number of times the same phrase is repeated in the train- ing set, “Epochs” is the number of training iterations the model undergoes during the update process.
  • The Complexity Paradox: While complex, high-entropy passwords take slightly longer to memorize than simple ones, they are still eventually fully memorized.
  • Vocabulary as a Leak: The timing attack is particularly dangerous because it doesn't require the attacker to know the "surrounding sentence" context of the secret. It purely exploits how the software manages its internal dictionary.
  • Attribute Inference: Even if the attacker doesn't guess the exact secret, the model's confidence scores for similar words (in terms of prefix, suffix, or shape) are often higher, allowing an attacker to "hill-climb" toward the actual secret.

Key Results

  • Memorization Attack: Achieved a 70% AUC (Area Under the Curve) on a practical medical document redaction use case.
  • Timing Attack: Achieved an overwhelming 99.23% AUC success rate. The difference in execution time between member and non-member words was consistent enough to be a highly reliable signal.
  • Version Comparison: The researchers noted that spaCy v3.0.3 (which uses the architecture shown in Figure 1/4) memorizes data significantly faster than the older v2.3.2.
Figure 1: SpaCy’s NER Model. An input sentence is broken up into words which are individually processed into an em- bedding (shaded green). This embedding is then converted into a contextual Encoding, by combining the current word embedding with the embeddings of surrounding sentence words (shaded blue), before being processed by the Multi- Layer Perceptron.

Limitations and Future Work

The authors suggest several defenses, though each has trade-offs:

  • Differential Privacy (DP): While effective at stopping neural memorization, it can be difficult to implement and may reduce model accuracy.
  • Constant Time Delays: Adding a delay to queries to mask the timing difference can thwart the side-channel attack but slows down the service for all users.
  • Synthetic Data: Replacing real secrets with synthetic ones during training helps, but if the synthetic data doesn't perfectly match the distribution of real data, the model's utility for redaction drops significantly.

Future work is needed to develop NER architectures that are "privacy-by-design," ensuring that the process of learning to protect data doesn't create a new way to leak it.

Summary updated

Abstract

Named entity recognition models (NER), are widely used for identifying named entities (e.g., individuals, locations, and other information) in text documents. Machine learning based NER models are increasingly being applied in privacy-sensitive applications that need automatic and scalable identification of sensitive information to redact text for data sharing. In this paper, we study the setting when NER models are available as a black-box service for identifying sensitive information in user documents and show that these models are vulnerable to membership inference on their training datasets. With updated pre-trained NER models from spaCy, we demonstrate two distinct membership attacks on these models. Our first attack capitalizes on unintended memorization in the NER's underlying neural network, a phenomenon NNs are known to be vulnerable to. Our second attack leverages a timing side-channel to target NER models that maintain vocabularies constructed from the training data. We show that different functional paths of words within the training dataset in contrast to words not previously seen have measurable differences in execution time. Revealing membership status of training samples has clear privacy implications, e.g., in text redaction, sensitive words or phrases to be found and removed, are at risk of being detected in the training dataset. Our experimental evaluation includes the redaction of both password and health data, presenting both security risks and privacy/regulatory issues. This is exacerbated by results that show memorization with only a single phrase. We achieved 70% AUC in our first attack on a text redaction use-case. We also show overwhelming success in the timing attack with 99.23% AUC. Finally we discuss potential mitigation approaches to realize the safe use of NER models in light of the privacy and security implications of membership inference attacks.
BibTeX
@misc{ali2022unintended,
  author = {Rana Salal Ali and Benjamin Zi Hao Zhao and Hassan Jameel Asghar and Tham Nguyen and Ian David Wood and Dali Kaafar},
  title = {{Unintended Memorization and Timing Attacks in Named Entity Recognition Models}},
  year = {2022},
  eprint = {2211.02245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.02245},
}
Figure