Abstract
Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differential Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks.
Introduction
Large Language Models (LLMs) are increasingly trained on massive datasets that may contain sensitive personal identifiable information (PII). While previous research has demonstrated that auto-regressive generative models like GPT-2 can be prompted to reveal verbatim training data, less is known about encoder-only models like BERT. BERT models are widely used in industrial NLP for classification tasks, often fine-tuned on private data such as internal emails or customer support logs. This study addresses whether fine-tuning BERT on sensitive datasets leads to the "leakage" of private information through named entity memorization.
Methodology
The researchers evaluated three distinct fine-tuning configurations for a text classification task to observe their impact on privacy and performance:
- Full Fine-tuning: All layers of the BERT model are updated during training.
- Partial Fine-tuning: Most layers are frozen, with only the final encoding layer and the classifier head being trained.
- Differentially Private (DP) Fine-tuning: This setup uses the DP-SGD (Differentially Private Stochastic Gradient Descent) algorithm to add noise to gradients during partial fine-tuning, providing a formal mathematical guarantee of privacy.
To evaluate data leakage, the authors developed an extraction pipeline. Since BERT is not natively designed for long-form text generation, they implemented a sequential sampling strategy. This involved providing a "seed" prompt and iteratively masking and predicting subsequent tokens. Two prompting strategies were tested: Naive Prompting (using random web snippets) and Informed Prompting (using snippets from the private dataset's test set).
The generated text samples were then processed using Named Entity Recognition (NER) to identify entities, which were cross-referenced against the original training data to determine a "memorization rate."
Experimental Setup
The study utilized two benchmark datasets containing naturally occurring PII:
- Enron Email Dataset: Real emails from Enron employees, categorized by folder.
- Blog Authorship Corpus: Blog posts labeled by topic.
To isolate entities unique to the fine-tuning data, the authors used the original BERT pre-training sets (BookCorpus and Wikipedia) to filter out common entities.
Results and Discussion
The experiments yielded several key insights into BERT's privacy characteristics:
- Inherent Privacy Resilience: BERT's auto-encoder architecture and smaller parameter count compared to GPT-style models make it significantly less prone to memorizing long sequences. Fine-tuned models did not leak substantially more entities than the original "Base" model.
- Effectiveness of Naive Attacks: Surprisingly, random web prompts (Naive) were more effective at triggering entity generation than Informed prompts. This suggests that attackers do not require specific knowledge of the training data to attempt extraction.
- The Cost of Differential Privacy: While DP effectively reduced the memorization rate to near zero (1.4% on Enron, ~0% on Blogs), it severely degraded the model's text generation coherence and classification accuracy. On the Enron dataset, DP fine-tuning resulted in a 16 percentage point drop in accuracy compared to standard partial fine-tuning.
- Entity Complexity: Extracted entities were primarily simple types like locations. Complex entities, such as specific monetary amounts, were rarely memorized.
Across all setups, the memorization rate for private entities remained below 10%. Interestingly, on the Enron dataset, the "Full" fine-tuned model showed a lower extraction rate (5.5%) than the Base model (9.3%), suggesting that fine-tuning might overwrite some pre-trained information.
Conclusion
This study demonstrates that BERT is relatively resistant to training data extraction attacks compared to generative models. While Differential Privacy offers strong protection, its impact on model utility is significant. Future research should explore these dynamics in non-English datasets and investigate the vulnerability of embedding layers to membership inference attacks.
Summary updated