Abstract
Differentially private synthetic data is a promising alternative for sensitive data release. Many differentially private generative models have been proposed in the literature. Unfortunately, they all suffer from the low utility of the synthetic data, particularly for images of high resolutions. Here, we propose DPAF, an effective differentially private generative model for high-dimensional image synthesis. Different from the prior private stochastic gradient descent-based methods that add Gaussian noises in the backward phase during the model training, DPAF adds a differentially private feature aggregation in the forward phase, bringing advantages, including the reduction of information loss in gradient clipping and low sensitivity for the aggregation. Moreover, as an improper batch size has an adverse impact on the utility of synthetic data, DPAF also tackles the problem of setting a proper batch size by proposing a novel training strategy that asymmetrically trains different parts of the discriminator. We extensively evaluate different methods on multiple image datasets (up to images of 128x128 resolution) to demonstrate the performance of DPAF.
Introduction: The Challenge of Private Image Synthesis
Training generative models, such as Generative Adversarial Networks (GANs), on sensitive datasets like medical records or private photographs carries a significant privacy risk. Models often "memorize" specific training examples, which can be extracted by malicious actors. Differential Privacy (DP) is the mathematical gold standard for mitigating this risk, but it typically incurs a heavy "utility cost," resulting in poor image quality.
Traditional approaches rely on DPSGD (Differentially Private Stochastic Gradient Descent), which injects noise into gradients during the backward pass of training. This process requires "clipping" gradients to a fixed size to bound sensitivity, which destroys fine-grained information. Additionally, GANs suffer from a "batch size dilemma": while DP noise is easier to manage with large batches, GAN discriminators often require small batches for optimal performance. Consequently, DP-generated images have historically been limited to low resolutions and high levels of blurriness.
Methodology: Shifting Noise to the Forward Pass
DPAF (Differentially Private Aggregation in Forward Phase) introduces a paradigm shift by moving the noise injection step from the training gradients to the model's forward pass. The approach is built on four key pillars:
- Forward Phase Aggregation (DPAGG): Instead of sanitizing the gradients of every layer, DPAF performs a differentially private aggregation of image features during the forward pass. By privatizing features early in the network, subsequent layers are automatically protected due to the "post-processing" property of DP. This allows later layers to be updated using standard, non-clipped SGD, preserving structural learning.
- Simplified Instance Normalization (SIN): To bound the sensitivity of the aggregation, DPAF uses a modified normalization technique that fixes the feature maps' $\ell_2$-norm to a predictable value ($\sqrt{mp}$). This provides a formal mathematical bound for privacy without the destructive effects of global gradient clipping.
- Asymmetric Training Strategy: To resolve the batch size dilemma, DPAF updates different parts of the discriminator at different frequencies. The "noisy" components are updated using a "virtual" large batch accumulated over multiple iterations, while non-noisy components maintain agility with smaller batches.
- DP Transfer Learning: The model utilizes a feature extractor from a DP-trained classifier, providing the generator with a "head start" on understanding fundamental image structures like edges and textures.
Key Insights: The Power of Post-Processing
The core innovation of DPAF is leveraging the post-processing property of differential privacy. Once the features are aggregated with noise in the forward pass, the information is already "sanitized." This bypasses the need for the destructive clipping process for a significant portion of the model's parameters.
Furthermore, by aggregating features before adding noise, the Signal-to-Noise Ratio (SNR) is significantly improved. The "signal" (actual image data) becomes much stronger relative to the noise compared to adding noise to individual, high-dimensional gradients.
Results and Performance
DPAF was evaluated on standard benchmarks including MNIST, Fashion-MNIST, CelebA, and FFHQ. Key findings include:
- Visual Breakthrough at High Resolution: At 128x128 resolution (FFHQ), where previous state-of-the-art (SOTA) methods like PEARL produced only random static, DPAF generated recognizable human faces with distinct features such as eyes, lips, and face shapes.
- Superior Accuracy: On the CelebA-Gender dataset with a strict privacy budget ($\epsilon = 1$), DPAF achieved 80.2% classification accuracy, a 14.57% improvement over previous SOTA.
- Quantitative Quality: DPAF consistently achieved lower (better) Fréchet Inception Distance (FID) scores, indicating that its synthetic images are statistically much closer to real-world data than those produced by competing DP models.
Conclusion and Future Work
DPAF demonstrates that moving DP noise injection to the forward phase can significantly improve the utility of generative models. While it represents a major step forward for 128x128 resolution synthesis, scaling to megapixel resolutions remains a challenge. Future work may involve integrating modern GAN architectures, such as residual blocks, to further enhance image fidelity.
Summary updated