Abstract
This paper introduces Differentially Private Variational Inference (DPVI), a general-purpose framework for performing private Bayesian inference on complex, non-conjugate models. By combining gradient clipping and noise injection with doubly stochastic variational inference, the authors achieve predictive accuracy nearly identical to non-private methods while maintaining rigorous differential privacy guarantees. The approach leverages the Moments Accountant to optimize the privacy-utility trade-off, significantly outperforming existing sampling-based methods like DP-SGLD.
Introduction
Bayesian inference is a powerful tool for understanding uncertainty in sensitive datasets, such as medical records or consumer behavior. However, standard inference algorithms can inadvertently leak information about individual data points. Differential Privacy (DP) provides a mathematical framework to ensure that the output of an algorithm does not reveal whether a specific individual was included in the dataset.
While DP versions of some Bayesian methods exist, they typically suffer from two main drawbacks. First, they are often limited to "conjugate" models, which are mathematically convenient but do not cover many real-world scenarios. Second, sampling-based methods like Markov Chain Monte Carlo (MCMC) are computationally expensive. In a DP setting, every iteration consumes part of a fixed privacy budget. MCMC often requires too many iterations to converge before the privacy budget is exhausted, leading to poor predictive results.
Methodology
DPVI addresses these challenges by treating inference as an optimization problem rather than a sampling problem. It builds on Doubly Stochastic Variational Inference (DSVI), which uses stochastic gradient ascent to approximate the posterior distribution.
To make this process differentially private, the authors implement three key steps:
- Gradient Clipping: To bound the influence of any single individual, the gradient contribution of each data point is clipped so its magnitude (norm) never exceeds a threshold ($c_t$).
- Gaussian Noise Injection: Random Gaussian noise is added to the sum of these clipped gradients to mask individual contributions.
- Moments Accountant: The authors use the Moments Accountant technique to track cumulative privacy loss ($\epsilon$) much more tightly than standard methods. This enables more training iterations for the same privacy cost.
The algorithm also utilizes privacy amplification through subsampling, where the randomness of mini-batch selection provides an inherent layer of privacy protection.
Experimental Results
The authors evaluated DPVI on binary classification tasks using the Abalone and Adult datasets, as well as a synthetic Gaussian Mixture Model (GMM) to test non-conjugate likelihoods.
As shown in the results, DPVI-MA (green line) reaches the performance of the non-private baseline (red dashed line) at very low $\epsilon$ values. In contrast, the sampling-based DP-SGLD (blue line) struggles to reach comparable accuracy even with a significantly higher privacy budget. Furthermore, in the GMM task, DPVI was able to recover the underlying data distribution much more accurately than sampling alternatives.
Key Insights
- Optimization vs. Sampling: Variational inference (optimization) is fundamentally better suited for DP than MCMC. A good optimizer can reach a solution in fewer steps than a sampler needs to explore a distribution, preserving the limited privacy budget more effectively.
- Handling Latent Variables: For models with latent variables, the authors suggest marginalizing them out to avoid the need to privatize individual-specific parameters, which would otherwise require impractical amounts of noise.
Limitations and Future Work
Despite its success, DPVI is sensitive to hyperparameters such as the clipping threshold ($c_t$) and the subsampling rate ($q$).
As seen above, the choice of $q$ significantly impacts the privacy-utility trade-off. In a real-world private setting, tuning these parameters is challenging because standard validation techniques also consume the privacy budget. Developing automated, robust heuristics for choosing these thresholds remains a critical area for future research.
Summary updated