Differentially Private Next-Token Prediction of Large Language Models

James Flemings, Meisam Razaviyayn, Murali Annavaram

Abstract

Large Language Models (LLMs) are prone to memorizing sensitive training data, creating privacy vulnerabilities. While Differentially Private Stochastic Gradient Descent (DP-SGD) is the standard approach for privacy-preserving training, it faces significant challenges in efficiency and scalability for large-scale models. This paper presents PMixED (Private Mixing of Ensemble Distributions), a framework that shifts the privacy mechanism from the training phase to the prediction phase. By leveraging ensemble distributions and a novel mathematical technique called RD-mollification, PMixED provides robust privacy guarantees with lower computational overhead and better utility than traditional private training methods.

Introduction

The "memorization" problem in LLMs allows adversaries to potentially extract sensitive information through targeted queries. The current gold standard for privacy, DP-SGD, requires per-sample gradient calculations that are memory-intensive and slow. Furthermore, the noise added in DP-SGD scales with model parameters, making it less effective for massive LLMs. PMixED addresses these issues by assuming a "black-box" access model—common in commercial APIs—and providing privacy at the token-prediction level.

The PMixED Framework

PMixED operates in two distinct phases: training an ensemble of experts and mixing their predictions privately.

Phase I: Training

The private dataset is partitioned into $N$ disjoint subsets. A pre-trained LLM is fine-tuned on each subset using Low-Rank Adaptation (LoRA). This creates an ensemble of "experts," where each model is exposed to only a small fraction of the sensitive data.

Phase II: Prediction

During inference, PMixED processes a query $x_t$ through the following steps:

  1. Poisson Subsampling: A random subset of the ensemble is selected.
  2. RD-Mollification: The output distribution of each selected expert is projected onto a Renyi Divergence (RD) ball centered around a public model's distribution (e.g., a base GPT-2).
  3. Mixing and Aggregation: A mixing parameter blends the private expert knowledge with the public baseline. The final distribution is averaged across the ensemble.
  4. Sampling: The next token is sampled directly from this "mollified" distribution.
Overview of the PMixED framework, illustrating the transition from partitioned training (Phase I) to private inference and projection (Phase II).

Experimental Results

The authors evaluated PMixED on the WikiText-103 and One Billion Word (LM1B) benchmarks using GPT-2 small as the public reference.

At a privacy budget of $\epsilon = 8$, PMixED consistently outperformed DP-SGD in terms of perplexity (PPL). On WikiText-103, PMixED achieved a PPL of 31.95, compared to 32.96 for DP-SGD. On the larger One Billion Word dataset, the improvement was even more pronounced, demonstrating the scalability of the prediction-time approach.

Comparison of Perplexity (PPL) across different training and privacy methods. PMixED achieves lower perplexity than DP-SGD on both WikiText-103 and One Billion Word datasets.

Key Insights

  • Inherent Stochasticity: Unlike DP-SGD which adds external Gaussian noise, PMixED utilizes the existing randomness in token sampling to satisfy differential privacy.
  • Model Agnosticism: Because the privacy mechanism operates on output distributions, it is independent of the model's internal architecture, making it suitable for black-box API deployments.
  • Efficiency via LoRA: The use of LoRA adapters makes maintaining an ensemble of 80+ models computationally feasible, as the base weights remain shared.

Conclusion

PMixED represents a significant step toward practical privacy for LLMs. By moving the privacy boundary to the prediction phase, it bypasses the hardware bottlenecks of DP-SGD while maintaining high text quality. Future work will focus on reducing inference latency and extending the privacy proofs to more complex decoding strategies like beam search.

Summary updated

Abstract

Ensuring the privacy of Large Language Models (LLMs) is becoming increasingly important. The most widely adopted technique to accomplish this is DP-SGD, which trains a model to guarantee Differential Privacy (DP). However, DP-SGD overestimates an adversary’s capabilities in having white box access to the model and, as a result, causes longer training times and larger memory usage than SGD. On the other hand, commercial LLM deployments are predominantly cloud-based; hence, adversarial access to LLMs is black-box. Motivated by these observations, we present Private Mixing of Ensemble Distributions (PMixED): a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-token sampling and a public model to achieve Differential Privacy. We formalize this by introducing RD-mollifers which project each of the model’s output distribution from an ensemble of fine-tuned LLMs onto a set around a public LLM’s output distribution, then average the projected distributions and sample from it. Unlike DP-SGD which needs to consider the model architecture during training, PMixED is model agnostic, which makes PMixED a very appealing solution for current deployments. Our results show that PMixED achieves a stronger privacy guarantee than sample-level privacy and outperforms DP-SGD for privacy 𝜖 = 8 on large-scale datasets. Thus, PMixED offers a practical alternative to DP training methods for achieving strong generative utility without compromising privacy.
BibTeX
@inproceedings{flemings2024differentially,
  author = {James Flemings and Meisam Razaviyayn and Murali Annavaram},
  title = {{Differentially Private Next-Token Prediction of Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=a3xHgjVfRn},
}
Figure