Abstract
Recent advances in machine learning have largely benefited from the massive accessible training data. However, large-scale data sharing has raised great privacy concerns. In this work, we propose a novel privacy-preserving data Generative model based on the PATE framework (G-PATE), aiming to train a scalable differentially private data generator that preserves high generated data utility. Our approach leverages generative adversarial nets to generate data, combined with private aggregation among different discriminators to ensure strong privacy guarantees. Compared to existing approaches, G-PATE significantly improves the use of privacy budgets. In particular, we train a student data generator with an ensemble of teacher discriminators and propose a novel private gradient aggregation mechanism to ensure differential privacy on all information that flows from teacher discriminators to the student generator. In addition, with random projection and gradient discretization, the proposed gradient aggregation mechanism is able to effectively deal with high-dimensional gradient vectors. Theoretically, we prove that G-PATE ensures differential privacy for the data generator. Empirically, we demonstrate the superiority of G-PATE over prior work through extensive experiments. We show that G-PATE is the first work being able to generate high-dimensional image data with high data utility under limited privacy budgets ($\epsilon \le 1$).
Introduction
Machine learning thrives on massive datasets, but sharing sensitive data—like medical records or personal photos—poses significant privacy risks. While Generative Adversarial Networks (GANs) can create synthetic versions of this data, standard GANs offer no formal privacy guarantees; they might inadvertently "memorize" and leak individual training examples.
Existing solutions like DP-GAN (which uses DP-SGD) often suffer from a massive drop in utility because the noise added to every gradient update quickly exhausts the "privacy budget." Another approach, PATE-GAN, uses an ensemble of teachers to label synthetic data for a student, but it struggles to scale because the student discriminator never sees real data, making it difficult to learn complex, high-dimensional distributions like human faces. G-PATE addresses these scalability and utility issues by rethinking the privacy bottleneck in generative modeling.
Methodology
G-PATE (Generative PATE) rethinks how privacy is applied to the GAN training loop. The core philosophy is that the discriminator itself does not need to be private; only the information flowing from the discriminator to the generator must be protected.
The architecture consists of three main components:
- Ensemble of Teacher Discriminators: The sensitive training data is split into $n$ disjoint subsets. Each subset trains a standard, non-private "teacher" discriminator.
- Student Generator: Instead of a student discriminator, G-PATE uses a student generator that learns directly from the teachers' feedback.
- DPGradAgg (Private Gradient Aggregation): To update the generator, G-PATE calculates the gradients from all teacher discriminators. To keep this private, it uses a three-step aggregation: random projection to reduce dimensionality, gradient discretization into "bins," and the Confident-GNMax Aggregator which only releases the "winning" gradient direction if there is a strong consensus among teachers.
By using Rényi Differential Privacy (RDP), the authors provide a tight theoretical bound on the privacy leakage over thousands of training iterations. This "privacy through consensus" ensures that released gradients represent general features of the data distribution rather than individual records.
Experimental Results
The authors evaluated G-PATE on a mix of tabular and image datasets, including Kaggle Credit, MNIST, Fashion-MNIST, and CelebA. G-PATE significantly outperformed existing baselines across the board.
On MNIST with a very strict budget of $\epsilon = 1$, G-PATE achieved 58.8% classification accuracy on synthetic data, compared to just ~40% for DP-GAN and PATE-GAN. Furthermore, G-PATE is the first DP generative model to produce usable $64 \times 64$ images of faces under $\epsilon \le 1$. On the Kaggle Credit tabular dataset, G-PATE achieved an AUROC of 0.9072, coming remarkably close to the non-private vanilla GAN (0.9417).
Conclusion
G-PATE demonstrates that high-dimensional generative modeling is possible under strict differential privacy constraints. By focusing the privacy budget on the information flow between teacher discriminators and a student generator, G-PATE achieves state-of-the-art utility. Future work may explore different projection methods or more advanced generator architectures to further close the gap between private and non-private synthetic data.
Summary updated