Abstract
Many machine learning applications are based on data collected from people, such as their tastes and behaviour as well as biological traits and genetic data. Regardless of how important the application might be, one has to make sure individuals' identities or the privacy of the data are not compromised in the analysis. Differential privacy constitutes a powerful framework that prevents breaching of data subject privacy from the output of a computation. Differentially private versions of many important Bayesian inference methods have been proposed, but there is a lack of an efficient unified approach applicable to arbitrary models. In this contribution, we propose a differentially private variational inference method with a very wide applicability. It is built on top of doubly stochastic variational inference, a recent advance which provides a variational solution to a large class of models. We add differential privacy into doubly stochastic variational inference by clipping and perturbing the gradients. The algorithm is made more efficient through privacy amplification from subsampling. We demonstrate the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees, clearly improving over previous sampling-based alternatives especially in the strong privacy regime.
Introduction
Bayesian inference is a powerful tool for understanding uncertainty, but when the underlying data is sensitive (like medical records or personal habits), we need mathematical guarantees that the model's output won't leak individual information. Differential Privacy (DP) provides this guarantee. However, existing DP Bayesian methods face significant challenges: they are often limited to exponential family models, suffer from efficiency issues in MCMC-based approaches where every iteration consumes the privacy budget, and struggle with non-conjugate models common in real-world applications. This work introduces a unified, efficient approach applicable to almost any differentiable model.
Methodology: DPVI
The proposed method, Differentially Private Variational Inference (DPVI), builds upon Automatic Differentiation Variational Inference (ADVI). Variational Inference (VI) transforms inference into an optimization problem, finding parameters of a simple distribution (e.g., Gaussian) that approximate the true posterior.
To ensure Differential Privacy, DPVI incorporates several key mechanisms:
- Gradient Clipping: Gradients for individual data points in a mini-batch are clipped to a threshold $c_t$, preventing any single individual from disproportionately influencing model updates.
- Gaussian Mechanism: Calibrated Gaussian noise is added to the sum of clipped gradients.
- Privacy Amplification: The method leverages the privacy benefits of data subsampling via mini-batches.
- Moments Accountant: A sophisticated tracking technique for privacy loss, allowing for more optimization iterations under tighter privacy guarantees compared to standard composition theorems.
Experimental Evaluation
The authors evaluated DPVI on three main tasks: binary classification on the Abalone and Adult (census income) datasets from UCI, and a synthetic Gaussian Mixture Model (GMM) to test handling of latent variables and multi-modal distributions.
Key Results
As shown in the figure above, the proposed DPVI method (specifically the version using the Moments Accountant, DPVI-MA) significantly outperforms DP-SGLD, a popular private MCMC-like method. DPVI reaches higher accuracy more rapidly and at a lower privacy cost ($\epsilon$). For logistic regression tasks, DPVI achieves performance remarkably close to the non-private baseline even under strong privacy guarantees ($\epsilon < 0.5$). In GMM experiments, DPVI successfully recovered cluster parameters where DP-SGLD struggled.
The authors argue that VI is fundamentally better suited for DP than MCMC because optimization can reach a satisfactory solution with fewer noisy evaluations than a sampler requires for convergence. Furthermore, marginalizing latent variables is crucial; attempting to privatize them directly introduces excessive noise.
Discussion and Limitations
While DPVI is robust, it is sensitive to hyperparameters, particularly the clipping threshold $c_t$. The choice of $c_t$ significantly impacts test accuracy, as shown in the following figure.
In a strictly private setting, tuning $c_t$ is challenging as it consumes the privacy budget. Additionally, aggressive clipping shifts the stationary point of the optimization, meaning the private solution solves a slightly modified objective. Future work includes developing heuristics for $c_t$ selection and exploring dimensionality reduction to facilitate easier gradient clipping in high-dimensional spaces.
Conclusion
DPVI provides a general-purpose framework for private Bayesian inference. By combining doubly stochastic VI with gradient perturbation and the moments accountant, it achieves high accuracy for complex, non-conjugate models while maintaining strong privacy guarantees.
Summary updated