A Unified View of Differentially Private Deep Generative Modeling

Dingfan Chen, Raouf Kerkouche, Mario Fritz

As privacy regulations like GDPR and ADPPA tighten, the demand for high-quality synthetic data—data that mimics real distributions while providing formal privacy guarantees—has grown significantly. However, the research community has developed these models using a fragmented array of architectures (GANs, VAEs, Diffusion), privacy mechanisms (DP-SGD, PATE), and evaluation metrics. This paper by Dingfan Chen, Raouf Kerkouche, and Mario Fritz provides a systematic framework to unify this landscape.

The Unified Framework and the Privacy Barrier

The core challenge in Differentially Private Deep Generative Modeling (DP-DGM) is the fundamental tension between the high-dimensional data requirements of deep learning and the exponential difficulty of maintaining Differential Privacy (DP) as dimensionality increases. To address this, the authors introduce a taxonomy based on the "privacy barrier"—the specific point in the training pipeline where data is sanitized.

The general pipeline of differentially private deep generative modeling, illustrating the flow between the generator, synthetic data, measurement, and real data.

The authors propose four distinct positions for this barrier:

  • B1: Between Real Data and Measurement: These methods (e.g., DP-Merf) sanitize data features or embeddings in a "one-shot" fashion. They are stable and fast but may lack the expressiveness of learnable models.
  • B2: Within Measurement: This involves training a private "critic" or discriminator (e.g., DP-GAN). The generator never sees real data directly, only the sanitized output of the critic.
  • B3: Between Measurement and Synthetic Data: Here, a non-private critic provides a feedback signal (gradient) that is sanitized before reaching the generator (e.g., GS-WGAN). This allows for sharper guidance at a higher privacy cost.
  • B4: Within Generator: Standard for explicit density models like DP-VAEs or DP-Diffusion, where privacy mechanisms are integrated directly into the internal parameter updates.

Categorizing Generative Architectures

The landscape of DP-DGM can be further understood by how different generative architectures interact with these optimization strategies. Models are generally split between explicit density models (which define a likelihood) and implicit density models (which learn to sample without an explicit likelihood).

Taxonomy of generative models categorized by density type and optimization strategy (learnable vs. static critics).

As shown in the taxonomy above, implicit density models like GANs typically rely on learnable critics (B2/B3), while distribution matching techniques often use static critics (B1). Explicit density models like VAEs and Diffusion models typically fall under the B4 category, where the generator itself is trained privately.

Key Insights and Trade-offs

The "One-Shot" Advantage

Methods in category B1 (sanitizing mean embeddings) are surprisingly effective. By sanitizing the data only once, they avoid the "privacy leakage" that accumulates over thousands of training iterations in iterative methods like DP-SGD.

The Feedback Loop Trade-off

Category B3 allows for the use of state-of-the-art, non-private discriminators. While this provides a much sharper "teaching signal" to the generator, the privacy cost scales with the batch size, making it a high-cost, high-reward strategy.

Privacy Notion Mismatch

The authors highlight a critical issue in the field: many papers compare "replace-one" DP to "add-or-remove-one" DP without adjustment. A naive comparison can result in a factor of 2 difference in the effective privacy budget ($\epsilon$), potentially invalidating many reported state-of-the-art results.

Conclusion and Future Directions

The paper concludes that while GANs (B2/B3) pioneered the field, explicit density models like DP-Diffusion (B4) are rapidly becoming the most promising path for high-fidelity, private image generation due to their superior training stability.

Future work remains in bridging the "auditing gap"—proving models are as private as claimed—and addressing distribution shifts when public data is used to "warm-start" private training. Additionally, the field is encouraged to move toward task-guided generation, balancing the trade-off between visual fidelity and the utility of the data for specific downstream applications.

Summary updated

BibTeX
@article{chen2024unified,
  author = {Dingfan Chen and Raouf Kerkouche and Mario Fritz},
  title = {{A Unified View of Differentially Private Deep Generative Modeling}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2024},
  url = {https://openreview.net/forum?id=wUXP0UGrxH},
}
Figure