Memorization of Named Entities in Fine-tuned BERT Models

Andor Diera, Nicolas Lell, Aygul Garifullina, Ansgar Scherp

Abstract

Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differential Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks.

Introduction

As Large Language Models (LLMs) are increasingly integrated into commercial services, the risk of training data extraction—where a model inadvertently reveals sensitive information from its training set—has become a central concern. While auto-regressive models like the GPT family have been shown to leak verbatim training data when prompted, less is known about auto-encoder models like BERT.

BERT is frequently fine-tuned on small, domain-specific datasets that may contain Personally Identifiable Information (PII), such as corporate emails or medical records. This study investigates whether fine-tuning BERT for classification tasks causes it to "leak" private named entities through text generation, and whether defensive measures like Differential Privacy (DP) can mitigate these risks.

Methodology

To evaluate the privacy risks, the researchers developed a pipeline to extract and verify named entities from fine-tuned models.

Experimental pipeline for evaluating named entity memorization in BERT.

The experimental framework involves three distinct fine-tuning configurations to observe how different training depths and privacy constraints affect memorization:

  1. Full Fine-tuning: All model parameters, including all encoding layers and the classifier head, are updated.
  2. Partial Fine-tuning: Only the final encoding layer and the classifier head are updated, keeping the bulk of the model frozen.
  3. Differentially Private (DP) Fine-tuning: Utilizing the DP-SGD algorithm to add noise during the optimization process, providing a mathematical guarantee of privacy for individual data points.
The three fine-tuning configurations: Full, Partial, and Differentially Private.

Because BERT is not natively designed for text generation, the study employed a sequential sampling strategy. In this approach, the model predicts a "masked" token at the end of a sequence, which is then appended to the prompt to predict the subsequent token. Two prompting strategies were used: Naive Prompting (using random web snippets) and Informed Prompting (using snippets from the sensitive dataset's test set).

Datasets

The study utilized two primary benchmark datasets containing PII:

  • Enron Email Dataset: Real-world emails from Enron employees, used for a 7-class classification task.
  • Blog Authorship Corpus: Over 400,000 blog posts labeled by topic across 39 classes, containing names, locations, and personal anecdotes.

Key Insights and Results

The Baseline Surprise

A critical finding was that fine-tuned models did not generate significantly more private entities than the original pre-trained only base model. This suggests that many "leaked" entities (like common names) were likely learned during the initial massive pre-training on public data (Wikipedia/BooksCorpus) rather than the private fine-tuning set.

Prompting and Architecture

Counter-intuitively, Naive Prompts were more effective at triggering entity extraction than Informed Prompts. Random web snippets provided more diverse starting points, allowing the model to explore a wider range of its memorized space. Furthermore, BERT’s bidirectional nature and relatively small parameter count appear to make it inherently more resistant to verbatim memorization compared to larger generative models.

The DP Trade-off

While Differential Privacy (DP) successfully reduced memorization rates to near zero in some cases, it had a "detrimental effect" on the model's utility. In the Enron dataset, accuracy dropped from 86.8% (Full) to 68.3% (DP). More importantly, DP essentially "broke" the model's ability to generate coherent text, resulting in gibberish samples.

Memorization Rates

Across all experimental setups, the rate of successfully extracted private entities remained remarkably low, consistently staying below 10%. The most commonly leaked categories were "Location" and "Geo-Political Entity" (GPE), which are generally less unique and sensitive than specific names or financial values.

Conclusion and Future Work

The study concludes that BERT-based services are relatively resilient to training data extraction attacks targeting named entities. However, the researchers note that this study was limited to English and focused primarily on encoding layers. Future research could explore whether embedding layers are more susceptible to different types of privacy leaks, such as membership inference attacks, or how different language structures might influence memorization patterns.

Summary updated

Abstract

Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differential Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks.
BibTeX
@inproceedings{diera2023memorization,
  author = {Andor Diera and Nicolas Lell and Aygul Garifullina and Ansgar Scherp},
  title = {{Memorization of Named Entities in Fine-tuned BERT Models}},
  booktitle = {CD-MAKE},
  year = {2023},
  doi = {10.1007/978-3-031-40837-3\_16},
}
Figure