Abstract
Ensuring the privacy of Large Language Models (LLMs) is becoming increasingly important. The most widely adopted technique to accomplish this is DP-SGD, which trains a model to guarantee Differential Privacy (DP). However, DP-SGD overestimates an adversary's capabilities by assuming white-box access to the model, resulting in longer training times and larger memory usage than standard SGD. In contrast, commercial LLM deployments are predominantly cloud-based, where adversarial access is restricted to black-box interactions. Motivated by these observations, we present Private Mixing of Ensemble Distributions (PMixED): a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-token sampling and a public model to achieve Differential Privacy. We formalize this by introducing RD-mollifiers, which project the output distributions from an ensemble of fine-tuned LLMs onto a set around a public LLM's output distribution. Unlike DP-SGD, which is tied to model architecture during training, PMixED is model-agnostic. Our results show that PMixED achieves a stronger privacy guarantee than sample-level privacy and outperforms DP-SGD for privacy $\epsilon = 8$ on large-scale datasets, offering a practical alternative for achieving strong generative utility without compromising privacy.
Introduction
As Large Language Models (LLMs) are increasingly integrated into applications involving sensitive data, the risk of "memorization"—where a model inadvertently stores and later reproduces snippets of its training data—has become a primary concern. The standard defense, Differentially Private Stochastic Gradient Descent (DP-SGD), provides rigorous guarantees but often at a high cost to both computational efficiency and model utility.
Researchers at USC have proposed a paradigm shift: moving the privacy mechanism from the training phase to the inference phase. Their protocol, PMixED (Private Mixing of Ensemble Distributions), leverages the black-box nature of modern API-based LLM deployments to provide robust privacy guarantees with significantly less overhead than traditional private training methods.
The Limitations of DP-SGD
While DP-SGD is the gold standard for privacy-preserving machine learning, it faces three significant hurdles in the context of LLMs:
- Computational Overhead: It requires per-sample gradient clipping and noise addition, which drastically slows down training and increases memory consumption.
- Noise Scaling: As models grow in parameter count, the amount of noise required to maintain privacy increases, often degrading the model's ability to generate coherent text.
- Pessimistic Threat Models: DP-SGD assumes an attacker has "white-box" access to the model's internal weights. However, most commercial deployments (e.g., ChatGPT, Claude) are "black-box," where users only interact with the model via generated text.
The PMixED Protocol
PMixED addresses these issues by shifting the privacy-preserving step to the prediction phase. The protocol operates in two distinct phases:
Phase I: Ensemble Training
Instead of training a single model on the entire private dataset, the dataset is partitioned into $N$ disjoint subsets. A pre-trained LLM is then fine-tuned on each subset using Low-Rank Adaptation (LoRA). This creates an ensemble of "expert" models. Because LoRA only updates a small fraction of the model's parameters, maintaining an ensemble of dozens of adapters is memory-efficient compared to maintaining full model copies.
Phase II: Private Prediction and RD-Mollifiers
During inference, PMixED generates the next token by following these steps:
- Subsampling: A subset of the ensemble is randomly selected via Poisson subsampling.
- Distribution Generation: Each selected expert model, along with a non-private "public" model, generates a probability distribution for the next token.
- RD-Mollification: This is the core innovation of the paper. Rather than adding random Gaussian or Laplacian noise—which can disrupt the linguistic structure of the output—the protocol uses (α, β)-RD Mollifiers. These mathematical operators project the private models' distributions toward the public model's distribution. This ensures that the final output does not deviate too far from a "safe" baseline, effectively muffling any unique signatures from the private training data.
- Aggregation and Sampling: The projected distributions are averaged, and the next token is sampled from this combined distribution.
Key Insights
Randomness for Free
LLMs are naturally stochastic; they already sample from probability distributions to generate text. PMixED recognizes that this inherent randomness can be repurposed to satisfy the requirements of Differential Privacy. By using the sampling process itself as the source of privacy-preserving noise, the model avoids the "utility tax" typically associated with additive noise.
Group-Level Privacy
By partitioning the data into an ensemble, PMixED naturally provides group-level privacy. This is a more robust guarantee than standard sample-level privacy, as it protects against attackers trying to identify specific users or clusters of data within the training set.
Experimental Results
The authors evaluated PMixED using GPT-2 on the WikiText-103 and One Billion Word (LM1B) benchmarks.
- Superior Utility: On WikiText-103, PMixED achieved a perplexity of 31.95, significantly outperforming the base pre-trained model (38.63) and the DP-SGD baseline.
- Privacy Performance: At a privacy budget of $\epsilon = 8$, PMixED demonstrated better text generation quality than DP-SGD.
- Efficiency: Because it avoids the complexities of per-sample gradients and utilizes LoRA, PMixED is faster to train and more practical for deployment on standard hardware.
Limitations and Future Work
Despite its advantages, PMixED has constraints. Like all DP methods, it operates under a "privacy budget"; once the budget is exhausted through token generation, the model must stop to prevent information leakage. Additionally, the current protocol is optimized for ancestral sampling. Other decoding strategies, such as "top-k" or greedy decoding, would require further mathematical refinement to maintain the same privacy guarantees. Finally, while LoRA is efficient, managing an ensemble of 80+ adapters for a single request remains a challenge for high-throughput systems, suggesting a need for further parallelization research.
Conclusion
PMixED represents a significant step toward practical, private LLMs. By focusing on the black-box deployment scenario and utilizing the inherent properties of next-token prediction, it offers a path toward high-utility generative AI that respects data privacy without the prohibitive costs of traditional DP training.
Summary updated