Abstract
We propose a differentially private data generation paradigm using random feature representations of kernel mean embeddings when comparing the distribution of true data with that of synthetic data. We exploit the random feature representations for two important benefits. First, we require a minimal privacy cost for training deep generative models. This is because unlike kernel-based distance metrics that require computing the kernel matrix on all pairs of true and synthetic data points, we can detach the data-dependent term from the term solely dependent on synthetic data. Hence, we need to perturb the data-dependent term only once and then use it repeatedly during the generator training. Second, we can obtain an analytic sensitivity of the kernel mean embedding as the random features are norm bounded by construction. This removes the necessity of hyper-parameter search for a clipping norm to handle the unknown sensitivity of a generator network. We provide several variants of our algorithm, differentially-private mean embeddings with random features (DP-MERF) to jointly generate labels and input features for datasets such as heterogeneous tabular data and image data. Our algorithm achieves drastically better privacy-utility trade-offs than existing methods when tested on several datasets.
1. Introduction
The proliferation of sensitive datasets in fields like healthcare and finance necessitates robust privacy-preserving data sharing methods. Differential Privacy (DP) has emerged as the gold standard for individual privacy protection. However, generating high-utility synthetic data under DP constraints remains challenging. Current state-of-the-art approaches primarily rely on Generative Adversarial Networks (GANs) trained with DP-Stochastic Gradient Descent (DP-SGD). These methods suffer from two primary drawbacks: a continuous "privacy leak" as the model repeatedly accesses real data during training, and the difficulty of tuning "clipping norms" to manage gradient sensitivity, which often destabilizes training.
2. Methods: DP-MERF
We introduce Differentially Private Mean Embeddings with Random Features (DP-MERF), a framework that shifts away from adversarial training. Instead, we utilize Maximum Mean Discrepancy (MMD) to measure the distance between the true data distribution and the synthetic distribution.
The core innovation lies in the use of Random Fourier Features (RFF) to create a "private sketch" of the dataset:
- One-Shot Privatization: We compute a mean embedding of the real dataset using RFFs. Because RFFs are norm-bounded (values between -1 and 1), we can analytically determine the sensitivity of this summary.
- Static Privacy Budget: We add noise to this summary once. Since the generator is subsequently trained to match this noisy summary without further access to the real data, the training process does not incur additional privacy costs, regardless of the number of iterations.
- Analytic Sensitivity: By knowing the exact sensitivity ($2/m$), we eliminate the need for heuristic hyperparameter searches for clipping norms, a common pain point in DP-GANs.
3. Experimental Evaluation
We evaluated DP-MERF against standard benchmarks, including 2D Gaussian mixtures, tabular datasets (e.g., Adult, Credit, Epileptic, Intrusion), and image datasets (MNIST, FashionMNIST).
4. Key Results and Discussion
DP-MERF demonstrates a superior privacy-utility trade-off. On MNIST, DP-MERF at a strict $\epsilon=0.2$ outperformed GAN-based methods even when they were allowed a much larger budget ($\epsilon=10$). In tabular data experiments, DP-MERF consistently achieved higher accuracy across various predictive models compared to DP-GAN and DP-CGAN.
A critical advantage of DP-MERF is its ability to preserve the full diversity of the data distribution. In Gaussian mixture experiments, DP-MERF successfully captured all modes of the data at $\epsilon=1$, whereas private GANs exhibited significant mode collapse, failing to represent several clusters.
Interestingly, while synthetic images from DP-MERF may appear blurrier to humans, they often provide higher utility for training downstream machine learning classifiers than the visually "cleaner" images produced by private GANs. This suggests that DP-MERF better captures the underlying statistical properties necessary for learning.
5. Limitations and Future Work
Despite its advantages, DP-MERF requires a large number of random features ($D \approx 10,000$) to capture complex distributions, which can be memory-intensive. Furthermore, performance is dependent on the choice of kernel. Future research will focus on optimizing kernel selection for highly complex data and improving the visual sharpness of generated images through more sophisticated architectural priors.
Summary updated