Memorization of Named Entities in Fine-Tuned BERT Models

Andor Diera, Nicolas Lell, Aygul Garifullina, Ansgar Scherp

TL;DR

Researchers investigated whether fine-tuning BERT on sensitive datasets (like private emails) causes it to "leak" private named entities through text generation. The study found that BERT is surprisingly resistant to these attacks: fine-tuned models rarely leaked more private information than the original pre-trained base model, and the overall memorization rate remained below 10%.

What Problem Does This Paper Address?

Large Language Models (LLMs) are often trained on massive, uncurated datasets that inevitably contain Personally Identifiable Information (PII)—names, addresses, and social security numbers. While previous research has shown that auto-regressive models like GPT-2 can be "prompted" to spit out training data verbatim (a training data extraction attack), less was known about auto-encoder models like BERT.

Because BERT is typically fine-tuned on smaller, domain-specific datasets (e.g., a company’s internal emails) for classification tasks, there is a significant privacy risk. If an attacker gains access to a fine-tuned BERT service, could they force it to generate and reveal the sensitive entities it saw during training?

Methods

The authors designed a pipeline to test the "leakiness" of BERT across three fine-tuning configurations:

  1. Full: All layers of the model are updated.
  2. Partial: Only the last encoding layer and the classifier head are updated.
  3. Differential Privacy (DP): Using the DP-SGD algorithm to add noise during training, theoretically masking the influence of any single data point.

To extract data, they had to overcome a hurdle: BERT isn't designed for text generation. They used a sequential sampling strategy, where the model predicts a "masked" token at the end of a sequence, which is then appended to the prompt to predict the next token.

They tested two prompting strategies:

  • Naive Prompting: Using random snippets from the web (Common Crawl).
  • Informed Prompting: Using snippets from the actual test set of the sensitive data.

Finally, they used Named Entity Recognition (NER) to identify entities in the generated text and cross-referenced them with the training sets to see if they were "memorized."

Datasets

The study utilized two benchmark datasets known to contain PII:

  • Enron Email Dataset: A collection of real-world emails from Enron employees, adapted for a 7-class classification task.
  • Blog Authorship Corpus: Over 400,000 blog posts labeled by topic (39 classes), containing names, locations, and personal anecdotes.

Key Insights and Ideas

  • The "Base" Baseline Surprise: One of the most striking findings was that the fine-tuned models didn't generate significantly more private entities than the pre-trained only base model. This suggests that when BERT outputs a name like "John Doe," it’s often because it learned that name during its massive initial pre-training on Wikipedia/BooksCorpus, not necessarily from the specific private dataset it was fine-tuned on.
  • Naive Prompts are Better Attackers: Counter-intuitively, random web prompts (naive) were more effective at triggering entity extraction than prompts from the same domain (informed). The authors suggest that naive prompts provide more diverse "starting points," allowing the model to explore a wider range of its memorized space.
  • Architecture as a Shield: BERT’s bidirectional nature and smaller parameter count (compared to the GPT family) seem to make it inherently more resistant to verbatim memorization. It struggles to maintain the long-term coherence required to leak complex private strings.
  • The DP Trade-off: While Differential Privacy (DP) successfully reduced memorization, it absolutely "broke" the model's text generation capabilities, resulting in gibberish. However, for classification tasks where generation isn't needed, DP remains a viable (though accuracy-reducing) defense.

Key Results

  • Memorization Rates: Across all setups and datasets, the rate of successfully extracted private entities was less than 10%.
  • Classification Performance: Full fine-tuning reached ~86.8% accuracy on Enron, while the DP version dropped to ~68.3%.
  • Entity Types: "Location" and "Geo-Political Entity" (GPE) were the most commonly leaked categories, likely because they are less unique than specific names or monetary values.
  • DP Effectiveness: In the Blog Authorship dataset, the DP model leaked almost zero entities, but its downstream task performance was significantly lower than the non-private versions.

Limitations and Future Work

The authors acknowledge several constraints:

  • Language Scope: The study was limited to English. Languages with different structures (like German's compound words) might yield different memorization patterns.
  • Generation Bias: The left-to-right sequential sampling used for extraction doesn't perfectly mirror how BERT "thinks" (which is bidirectional), potentially underestimating the model's true memorization.
  • Layer Depth: The study focused on the encoding layers. Future work could investigate if the embedding layers are more susceptible to different types of privacy leaks, such as membership inference attacks.

Summary updated

Abstract

Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differentially Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks (Source code and datasets are available at: https://github.com/drndr/bert_ent_attack. An extended version of this paper can be also found on arXiv [12]).
BibTeX
@inproceedings{diera2023memorization,
  author = {Andor Diera and Nicolas Lell and Aygul Garifullina and Ansgar Scherp},
  title = {{Memorization of Named Entities in Fine-Tuned BERT Models}},
  booktitle = {CD-MAKE},
  year = {2023},
  doi = {10.1007/978-3-031-40837-3\_16},
}
Figure