Abstract
Normalizing flows have shown to be a promising approach to deep generative modeling due to their ability to exactly evaluate density—other alternatives either implicitly model the density or use approximate surrogate density. In this work, we present a differentially private normalizing flow model for heterogeneous tabular data. Normalizing flows are in general not amenable to differentially private training because they require complex neural networks with larger depth (compared to other generative models) and use specialized architectures for which per-example gradient computation is difficult (or unknown). To reduce the parameter complexity, the proposed model introduces a conditional spline flow which simulates transformations at different stages depending on additional input and is shared among sub-flows. For privacy, we introduce two fine-grained gradient clipping strategies that provide a better signal-to-noise ratio and derive fast gradient clipping methods for layers with custom parameterization. Our empirical evaluations show that the proposed model preserves statistical properties of original dataset better than other baselines.
Introduction
Generating synthetic tabular data while maintaining privacy is a critical challenge in data science. Tabular datasets typically contain a mix of continuous (e.g., age, income) and categorical (e.g., occupation, race) variables, making them difficult to model accurately. While Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) are common choices, they often suffer from "mode collapse" or provide only approximate density estimates.
Normalizing Flows (NFs) offer an attractive alternative by allowing exact likelihood evaluation. However, applying Differential Privacy (DP) to NFs is non-trivial due to three main hurdles: the "cheating" behavior on discrete data (assigning infinite likelihood), the high parameter complexity which increases DP-SGD noise, and significant gradient imbalance across deep layers. This work introduces DP-HFlow, a specialized NF architecture designed to overcome these challenges.
Methodology: The DP-HFlow Architecture
DP-HFlow incorporates several technical innovations to enable efficient and private training on heterogeneous data.
Variational Dequantization
To handle discrete variables, the model employs variational dequantization. A conditional flow (LMCDF) maps discrete values into a continuous space, allowing the primary normalizing flow to operate on a unified continuous distribution.
Parameter-Sharing and Conditional Spline Flows
To mitigate the noise penalty associated with high parameter counts in DP training, the authors utilize a shared parameter network. Instead of independent weights for each layer, a single network generates parameters for multiple flow blocks, conditioned on a block index embedding. This significantly reduces the total number of parameters without sacrificing model expressivity.
Fine-grained Gradient Clipping
Standard DP-SGD uses "flat" clipping, which can drown out signals in layers with smaller gradient magnitudes. DP-HFlow introduces:
- Per-unit clipping: Adjusting thresholds for individual units within a layer based on their gradient magnitude.
- Stochastic Sparsification: Randomly zeroing out small gradients to improve the signal-to-noise ratio for the most significant updates.
Furthermore, the authors derived fast per-example gradient computation methods for specialized layers (e.g., LU-decomposition and low-rank transformations), making DP training computationally feasible.
Experimental Results
The model was evaluated on four benchmark datasets: Adult, Census, Covertype, and Intrusion.
Statistical Fidelity
DP-HFlow demonstrates superior performance in capturing marginal distributions compared to private baselines like DP-MERF. As shown in Figure 5, DP-HFlow's synthetic distributions closely match the real data across various features, whereas baselines often fail to capture the multi-modal or skewed nature of tabular columns. On the Covertype dataset, DP-HFlow achieved a Kendall’s $\tau$ RMSE of 0.0717, significantly outperforming the next best private baseline (0.1863).
Predictive Utility
Synthetic data generated by DP-HFlow maintains high utility for downstream tasks. Classifiers trained on DP-HFlow data consistently yielded higher Macro-F1 and AUROC scores. For instance, on the Adult dataset, it reached a Macro-F1 of 0.56, narrowing the gap to the non-private baseline (0.75) more effectively than previous DP methods.
Conclusion
DP-HFlow represents a significant step forward in private synthetic data generation. By leveraging the exact density estimation of Normalizing Flows and addressing the specific challenges of DP training through parameter sharing and fine-grained clipping, it achieves a superior balance of privacy and utility. Future work includes refining importance criteria for gradient clipping and further optimizing the trade-off between model complexity and privacy noise.
Summary updated