Understanding Unintended Memorization in Language Models Under Federated Learning

Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Francoise BeaufaysNAACL 2021

TL;DR

This paper investigates how Federated Learning (FL) impacts the unintended memorization of rare phrases in language models, a critical privacy concern. They discover that FL's inherent data clustering by user significantly reduces memorization, and further improvements can be achieved with Federated Averaging and differential privacy, all while maintaining model utility.

What Problem Does This Paper Address?

Large language models (LLMs), particularly those used for next-word prediction (NWP), have a tendency to memorize rare or unique sequences from their training data. This is problematic because LLMs are increasingly trained on sensitive user data, and such memorization can lead to privacy leaks. While traditional centralized learning methods have been studied for this issue, the rise of Federated Learning (FL) – where models are trained collaboratively across decentralized devices without raw data leaving the device – presents a new landscape. It's unclear how FL's unique characteristics, such as data being naturally partitioned by users, affect this memorization problem. This paper aims to formally study and understand the impact of FL's components on unintended memorization in NWP models and explore mitigation strategies.

Methods

The authors adapt the "secret sharer" framework, originally designed for centralized models, to the FL setting. This involves inserting "canary" phrases (out-of-distribution examples) into the training data and then evaluating how well the trained model memorizes them. Key methodological aspects include:

  • Federated Secret Sharer: They introduce parameters to control the probability of users being selected as "secret sharers" and the probability of their data being replaced by canaries. Poisson sampling is used for this selection and replacement.
  • Comparison of FL vs. Central Learning: They systematically compare NWP models trained under FL with those trained using traditional centralized learning (SGD on minibatches of records).
  • Investigating FL Components: The study isolates the effects of:
  • Data Clustering (Non-IID vs. IID): Comparing models trained on data naturally grouped by users (non-IID) versus randomly shuffled data (IID).
  • Learning Technique: Comparing SGD (centralized) with Federated Averaging (FL).
  • Effective Minibatch Size: Examining the impact of larger user groups in FL versus smaller record groups in SGD.
  • Differential Privacy (DP): Evaluating the effectiveness of user-level DP guarantees (DP-FedAvg) in mitigating memorization. They use the analytical moments accountant and Mironov's proposition for DP analysis.

Datasets

  • Stack Overflow Dataset: A modified version of the Stack Overflow dataset (hosted by TensorFlow Federated) was used. It contained 392K users and 93M records. For experiments, they created synthetic users by assigning data sequentially from shuffled records to simulate an IID setting.

Key Insights and Ideas

  • User Clustering is Key: The most significant factor reducing unintended memorization in FL is the natural clustering of data by users (non-IID data). This suggests that the structure of FL inherently provides a privacy benefit.
  • Federated Averaging Helps: Using the Federated Averaging optimizer, especially with larger effective minibatch sizes (more users per round), further reduces memorization compared to SGD.
  • DP-FedAvg Offers Strong Guarantees: Training with user-level differential privacy in FL provides robust protection against memorization, even with thousands of canary insertions spread across many users, while maintaining comparable model utility.
  • Privacy-Utility Trade-off: While increasing DP noise can further reduce memorization, it can also impact model utility. Finding the optimal balance is an ongoing challenge.

Key Results

  • Memorization Reduction: Models trained under FL (non-IID data) showed significantly less memorization than those trained in a centralized setting. For example, with FedAvg and 500 users, non-IID data resulted in 21 memorized canaries (RS method) compared to 66 for IID data.
  • DP-FedAvg Resilience: DP-FedAvg models were resilient to memorizing canaries even with thousands of insertions across over a hundred users, while providing comparable utility (accuracy ~23.7-24.6%, perplexity ~57.3-64.3).
  • Optimizer Impact: Using Adam as an optimizer decreased memorization but also noticeably reduced model utility compared to SGD. Momentum increased memorization.
  • Clipping Benefits: Even without adding noise, clipping user updates in DP-FedAvg reduced memorization, indicating its importance in the DP mechanism.

Limitations and Future Work

  • Secret Sharer Granularity: The current secret sharer attack operates at the record level. Future work could involve designing stronger attacks targeting data at the user granularity.
  • Client-Drift: While the study found comparable utility for non-IID data, the authors acknowledge that client-drift can generally lead to slower convergence and leave further exploration of this for future work.
  • Model Architecture and Dataset Dependence: The degree of unintended memorization can depend on the specific model architecture and dataset used.
  • Privacy-Utility Trade-off Optimization: Designing methods to improve the privacy-utility trade-offs in DP-FL is an interesting direction for future research.

Summary updated

BibTeX
@inproceedings{thakkar2021understanding,
  author = {Om Dipakbhai Thakkar and Swaroop Ramaswamy and Rajiv Mathews and Francoise Beaufays},
  title = {{Understanding Unintended Memorization in Language Models Under Federated Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure