Abstract
Large Language Models (LLMs) are prone to memorizing sensitive training data, creating privacy vulnerabilities. While Differentially Private Stochastic Gradient Descent (DP-SGD) is the standard approach for privacy-preserving training, it faces significant challenges in efficiency and scalability for large-scale models. This paper presents PMixED (Private Mixing of Ensemble Distributions), a framework that shifts the privacy mechanism from the training phase to the prediction phase. By leveraging ensemble distributions and a novel mathematical technique called RD-mollification, PMixED provides robust privacy guarantees with lower computational overhead and better utility than traditional private training methods.
Introduction
The "memorization" problem in LLMs allows adversaries to potentially extract sensitive information through targeted queries. The current gold standard for privacy, DP-SGD, requires per-sample gradient calculations that are memory-intensive and slow. Furthermore, the noise added in DP-SGD scales with model parameters, making it less effective for massive LLMs. PMixED addresses these issues by assuming a "black-box" access model—common in commercial APIs—and providing privacy at the token-prediction level.
The PMixED Framework
PMixED operates in two distinct phases: training an ensemble of experts and mixing their predictions privately.
Phase I: Training
The private dataset is partitioned into $N$ disjoint subsets. A pre-trained LLM is fine-tuned on each subset using Low-Rank Adaptation (LoRA). This creates an ensemble of "experts," where each model is exposed to only a small fraction of the sensitive data.
Phase II: Prediction
During inference, PMixED processes a query $x_t$ through the following steps:
- Poisson Subsampling: A random subset of the ensemble is selected.
- RD-Mollification: The output distribution of each selected expert is projected onto a Renyi Divergence (RD) ball centered around a public model's distribution (e.g., a base GPT-2).
- Mixing and Aggregation: A mixing parameter blends the private expert knowledge with the public baseline. The final distribution is averaged across the ensemble.
- Sampling: The next token is sampled directly from this "mollified" distribution.
Experimental Results
The authors evaluated PMixED on the WikiText-103 and One Billion Word (LM1B) benchmarks using GPT-2 small as the public reference.
At a privacy budget of $\epsilon = 8$, PMixED consistently outperformed DP-SGD in terms of perplexity (PPL). On WikiText-103, PMixED achieved a PPL of 31.95, compared to 32.96 for DP-SGD. On the larger One Billion Word dataset, the improvement was even more pronounced, demonstrating the scalability of the prediction-time approach.
Key Insights
- Inherent Stochasticity: Unlike DP-SGD which adds external Gaussian noise, PMixED utilizes the existing randomness in token sampling to satisfy differential privacy.
- Model Agnosticism: Because the privacy mechanism operates on output distributions, it is independent of the model's internal architecture, making it suitable for black-box API deployments.
- Efficiency via LoRA: The use of LoRA adapters makes maintaining an ensemble of 80+ models computationally feasible, as the base weights remain shared.
Conclusion
PMixED represents a significant step toward practical privacy for LLMs. By moving the privacy boundary to the prediction phase, it bypasses the hardware bottlenecks of DP-SGD while maintaining high text quality. Future work will focus on reducing inference latency and extending the privacy proofs to more complex decoding strategies like beam search.
Summary updated