TL;DR
This paper tackles the challenge of generating high-quality synthetic images while preserving differential privacy. They propose a novel method, DP-MEPF, that leverages pre-trained perceptual features from public data to significantly improve the privacy-utility trade-off, achieving state-of-the-art results on complex datasets like CIFAR-10 with strong privacy guarantees.
What Problem Does This Paper Address?
Generating realistic synthetic data, especially images, is a key goal in machine learning. However, when this generation needs to be differentially private (DP) to protect sensitive training data, existing methods like DP-SGD struggle. DP-SGD requires adding substantial noise at each training step, which severely degrades the quality of the generated samples, especially for complex datasets like CIFAR-10. Current DP generative models are often limited to simpler datasets like MNIST or require very large privacy budgets (high epsilon, meaning less privacy) to produce even mediocre results. This paper aims to bridge this gap by finding a way to generate high-quality images with strong privacy guarantees on more challenging datasets.
Methods
The core idea of DP-MEPF is to leverage the power of pre-trained feature extractors on large public datasets. Instead of training a DP generative model from scratch, they first train a feature extractor (like a VGG or ResNet) on a public dataset. These "perceptual features" capture rich information about images. Then, they use these features to define a kernel for calculating the Maximum Mean Discrepancy (MMD) between the distribution of the private data and the generated data. The key innovation is that the privacy mechanism (Gaussian noise) is applied once to the mean embedding of the private data's perceptual features, rather than at every optimization step as in DP-SGD. This significantly reduces the privacy cost and allows for much better utility. The generator is then trained to minimize this privatized MMD.
Datasets
- Public Datasets (for pre-training feature extractors): ImageNet, CIFAR-10, SVHN.
- Private Datasets (for generating synthetic data): MNIST, FashionMNIST, CelebA, CIFAR-10.
The paper uses standard, well-known datasets for both public and private data, allowing for direct comparison with prior work.
Key Insights and Ideas
- Perceptual Features are Key: The paper strongly advocates for using features learned from public data as a crucial component for effective DP data generation. These features act as a powerful representation that is less sensitive to the noise introduced for privacy.
- Privatize Once, Optimize Many: Applying the DP noise only once to the data-dependent term (the mean embedding of perceptual features) is far more efficient than injecting noise at every optimization step (as in DP-SGD). This is the main driver behind the improved utility.
- Public Data Domain Matters: The effectiveness of the public pre-training data depends on its complexity relative to the private data. More complex public datasets (like ImageNet) are more beneficial for generating complex private data (like CelebA) than simpler ones (like SVHN).
- MMD with Perceptual Features is a Good Loss: The MMD, when computed using perceptual features, serves as a stable and effective loss function for training the generator in a differentially private setting.
Key Results
- State-of-the-Art Performance: DP-MEPF achieves significantly better results than prior DP generative models on datasets like CIFAR-10, CelebA, and FashionMNIST, especially at low privacy budgets (e.g., $\epsilon \approx 2$).
- CIFAR-10 Generation: The method successfully generates CIFAR-10 level images with $\epsilon \approx 2$, a feat previously considered very difficult or impossible for DP methods.
- Improved Downstream Accuracy: Synthetic data generated by DP-MEPF leads to higher downstream classifier accuracies compared to other DP methods, indicating better data utility. For example, on MNIST with $\epsilon=10$, DP-MEPF achieves 90% MLP accuracy, compared to ~80% for other methods.
- High-Privacy Quality: Even at very low privacy budgets ($\epsilon=0.2$), DP-MEPF produces visually discernible samples for complex datasets, outperforming existing methods that often produce noisy or abstract outputs.
Limitations and Future Work
- Reliance on Public Data: The method's effectiveness is heavily dependent on the availability and quality of suitable public data for pre-training.
- Feature Extractor Choice: While VGG architectures showed strong performance, exploring lower-dimensional embeddings or pruning schemes could further reduce sensitivity and improve quality.
- Extending to Other Models: Applying the core idea to other generative models like GANs or VAEs with pre-trained components is a promising avenue.
- Domain Shift: The paper acknowledges the challenge of significant domain shift between public and private data, suggesting that more research is needed to address this.
- Fairness: The paper discusses the potential trade-off between differential privacy and fairness, noting that DP can disproportionately affect under-represented groups. While not directly addressed in their experiments, this is an important area for future work.
Summary updated