A self-attention-based differentially private tabular GAN with high data utility

Zijian Li, Zhihui Wang

TL;DR

DP-SACTGAN is a new framework for generating high-quality synthetic tabular data while strictly adhering to differential privacy (DP) standards. By combining self-attention mechanisms with a novel gradient-injection method called DP-HOOK, the authors manage to produce synthetic datasets that are far more useful for downstream machine learning tasks than previous privacy-preserving GANs.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are famous for creating realistic images, but they struggle with tabular data, which often contains a messy mix of continuous numbers (like salary) and discrete categories (like job title).

When you add Differential Privacy—a mathematical framework that ensures an individual's data cannot be reverse-engineered from the model—the problem gets harder. Standard DP techniques (like DP-SGD) inject noise into every part of the training process. While this protects privacy, it often "breaks" the model’s ability to learn complex patterns, leading to poor data utility (i.e., the synthetic data is too noisy to be useful for researchers). Furthermore, training DP-GANs is notoriously slow and unstable.

Methods

The authors introduce DP-SACTGAN, which stands out through three key innovations:

  1. DP-HOOK (Gradient Sanitization): Instead of the traditional approach of adding noise to every parameter in the Discriminator (which makes training unstable), the authors use a "hook" function. This function only injects noise into the gradients as they flow from the Discriminator back to the Generator. Since the Generator never sees the raw sensitive data, this targeted noise is enough to satisfy DP requirements while allowing the Discriminator to remain more accurate.
  2. Self-Attention Architecture: Tabular features are often related in non-linear ways. The model uses self-attention layers to help the GAN understand the global dependencies between different columns in a single row.
  3. Advanced Tabular Modeling: Borrowing from the CTAB-GAN+ architecture, the model uses a Variational Gaussian Mixture (VGM) to handle "mixed" data types—columns that act like both numbers and categories (e.g., a "borrowing" column where '0' means 'never borrowed' and any positive value is a continuous amount). It also uses a conditional vector to combat data imbalance, ensuring rare categories are represented in the synthetic output.

Datasets

The researchers evaluated their model on two well-known benchmarks:

  • Adult Dataset: A census-based dataset used for classification tasks (predicting if income exceeds $50k).
  • King Dataset: A dataset of house sales in King County, used for regression tasks (predicting house prices).

These datasets are standard for testing how well a generative model can replicate real-world statistical distributions.

Key Insights and Ideas

  • The "Generator-Only" Privacy Shield: A crucial realization is that the Generator is essentially a "post-processor." It only interacts with the sensitive data through the Discriminator's feedback. By sanitizing that feedback loop (the gradients) rather than the entire Discriminator, the authors achieve a better balance between privacy and performance.
  • Stability over Brute Force: Traditional DP-SGD is computationally expensive and often fails to converge. DP-HOOK provides a more stable training path, making it feasible to train deeper architectures like those using self-attention.
  • Layer Norm > Batch Norm: The authors found that Layer Normalization is more effective for tabular data because individual data points are relatively independent compared to pixels in an image.

Key Results

DP-SACTGAN consistently outperformed baselines like CTGAN, DP-GAN, and PATE-GAN across several metrics:

  • Machine Learning Utility: On the King dataset (Ridge regression), DP-SACTGAN achieved a Mean Absolute Error (MAE) of 0.011, significantly better than the next best DP model, CTAB-GAN+, which scored 0.041.
  • Statistical Accuracy: The model showed the lowest Wasserstein Distance (WD) on the King dataset, meaning its synthetic distribution was the closest to the real data.
  • Privacy Robustness: In Membership Inference Attack (MIA) experiments, DP-SACTGAN proved just as resilient as traditional DP methods, successfully preventing attackers from determining if a specific individual was in the training set.

Limitations and Future Work

While DP-SACTGAN excels at fitting continuous distributions, the authors noted that on the Adult dataset, its performance on discrete columns (measured by Jensen-Shannon Divergence) was slightly behind CTAB-GAN+. This suggests that while the self-attention mechanism is great for continuous data, there is still room to improve how the model handles purely categorical relationships.

Future work could involve refining the attention mechanism specifically for high-cardinality categorical features or exploring different noise-injection schedules to further boost utility at very low privacy budgets (low $\epsilon$).

Summary updated

Abstract

Generative Adversarial Networks (GANs) have become a ubiquitous technology for data generation, with their prowess in image generation being well-established. However, their application in generating tabular data has been less than ideal. Furthermore, attempting to incorporate differential privacy technology into these frameworks has often resulted in a degradation of data utility. To tackle these challenges, this paper introduces DP-SACTGAN, a novel Conditional Generative Adversarial Network (CGAN) framework for differentially private tabular data generation, aiming to surmount these obstacles. Experimental findings demonstrate that DP-SACTGAN not only accurately models the distribution of the original data but also effectively satisfies the requirements of differential privacy.
BibTeX
@misc{li2023self,
  author = {Zijian Li and Zhihui Wang},
  title = {{A self-attention-based differentially private tabular GAN with high data utility}},
  year = {2023},
  eprint = {2312.13031},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.13031},
}
Figure