Introduction
Training Latent Dirichlet Allocation (LDA) models on massive text datasets often creates a conflict between processing speed and data privacy. This paper introduces SUB-LDA, a framework that uses Poisson subsampling to simultaneously speed up training and "amplify" privacy protections. By strategically sampling only a fraction of the words in each document during training, the authors achieve significant efficiency gains while maintaining high model utility through a novel adaptive subsampling technique.
Latent Dirichlet Allocation (LDA) is a cornerstone of topic modeling, but it faces two major hurdles: computational expense when dealing with millions of documents (Efficiency) and the risk of exposing sensitive information in text data (Privacy). While Differential Privacy (DP) can protect this data by adding noise, it often degrades model utility and slows training. Existing private LDA methods, such as HDP-LDA, suffer from "privacy budget depletion" due to the high number of iterations required by Collapsed Gibbs Sampling (CGS).
Methods
The authors propose SUB-LDA, which integrates Poisson Subsampling into the CGS training loop. This approach leverages two main components:
Privacy Amplification
In DP, subsampling acts as a "force multiplier." By observing only a random subset of data, an observer's certainty about the influence of any specific sensitive word is reduced. Using Rényi Differential Privacy (RDP), the authors prove that subsampling significantly reduces privacy loss per iteration.
The Efficiency–Privacy Function ($I$)
A new metric, the efficiency-privacy function $I$, is introduced to evaluate the trade-off between speed and privacy. The authors demonstrate that an optimal subsampling ratio exists where both efficiency and privacy protection are maximized simultaneously.
Adaptive Subsampling (AS)
To mitigate utility loss from simple random sampling, the AS method identifies "frequent words" for specific topics and samples them at a higher rate. This preserves the topic "signal" while reducing the overall computational load.
Experimental Setup
The researchers evaluated SUB-LDA on three standard benchmarks:
- 20 Newsgroups: ~20,000 documents across 20 topics (~900k words).
- NIPS: Full papers from NIPS conferences (~1.9M words).
- ENRON: Emails from the Enron Corporation (~6.4M words).
Key Results and Insights
The study reveals that subsampling is a "win-win" for privacy and efficiency. Usually, privacy comes at the cost of efficiency; here, ignoring part of the data to save time naturally enhances privacy.
Speedup and Privacy Gains
SUB-LDA achieved significant speed improvements. On the ENRON dataset, reducing the subsampling ratio led to a near-linear increase in the speed-up ratio. Furthermore, under the same convergence conditions, SUB-LDA with a 0.1 subsampling ratio achieved a total privacy budget ($ε$) of 4.6, compared to 84 for the full-data baseline.
Utility Preservation
Using the Adaptive Subsampling (AS) method, the researchers maintained low perplexity—a measure of predictive performance—even when sampling only 10% of the words.
Conclusion
SUB-LDA demonstrates that strategic subsampling can effectively balance the competing demands of privacy, efficiency, and utility in LDA training. Future work may explore distributed or local DP settings to protect data at the source and further investigate hyperparameter sensitivity for real-world deployment.
Summary updated