Generate Synthetic Text Approximating the Private Distribution with Differential Privacy

Wenhao Zhao, Shaoyang Song, Chunlai Zhou

Abstract: Due to the potential leakage of sensitive information in text, there is a societal call for feeding privacy-preserving text to model training. Recently, a lot of work showed that using synthetic text with differential privacy, rather than private text, can provide a strong privacy protection. However, achieving higher semantic similarity between synthetic and private text has not been thoroughly investigated. In this paper, we propose an approach that combines the iteratively optimized mindset from genetic algorithms to align the distribution of synthetic text with that of private text. Furthermore, not only does the final synthetic text meet the requirements of privacy protection, but also has a high level of quality. Through comparisons with various baselines on different datasets, we demonstrate that our synthetic text can closely match the utility of private text, while providing privacy protection standards robust enough to resist membership inference attacks from malicious users.

Introduction

As Large Language Models (LLMs) become ubiquitous, sensitive data—such as medical records or private emails—is frequently used for model training or as "few-shot" examples in prompts. However, text contains unique linguistic patterns that can be exploited via Membership Inference Attacks (MIA) to identify individuals. Traditional solutions like text sanitization often fail to hide statistical signatures, while DP-SGD training is computationally expensive. This paper, presented at IJCAI-24, introduces a framework that generates synthetic text mimicking a private dataset's distribution while maintaining rigorous Differential Privacy (DP) guarantees.

Methodology

The core of the proposed method is an iterative approach inspired by Genetic Algorithms (GA). The pipeline operates in several stages:

  1. Preparation: A text decoder is trained on public data to map embeddings back into readable text, ensuring no private data is used in the model architecture itself.
  2. Private Selection: Using a LimitedDomain mechanism, "parent" samples are selected from a population based on their semantic proximity to the private dataset. DP noise is added during this selection to ensure individual records are not leaked.
  3. Offspring Generation: An Estimation of Distribution Algorithm (EDA) models the distribution of selected parents and samples new "offspring" embeddings. This allows the synthetic distribution to gradually drift toward the private distribution.
  4. Readability Filtering: To maintain high quality, candidates are filtered using GPT-2 perplexity scores to ensure the resulting text is coherent.
The iterative genetic algorithm framework for synthetic text generation.

Key Results

The researchers evaluated the framework on standard benchmarks including AGNews, Disaster, and Trec. A primary insight is the effectiveness of the iterative alignment. As shown in the performance metrics, the Wasserstein distance between the synthetic and private distributions steadily decreases over successive iterations, eventually outperforming established baselines like SentDP and DP-ICL.

Distribution distance between synthetic and private text over iterations compared to baselines.

In terms of utility, the synthetic text achieved impressive results in 6-shot in-context learning tasks. On the Disaster dataset with a privacy budget of $\epsilon=5$, the method reached 77.8% accuracy, nearly matching the non-private performance of 78.9%. Furthermore, Membership Inference Attacks (MIA) confirmed that the synthetic text provides robust protection, as attackers could not reliably distinguish between member and non-member records.

Conclusion

This framework demonstrates that distribution alignment is a powerful alternative to traditional data anonymization. By evolving synthetic data to match private distributions in the embedding space, the method achieves high utility for downstream tasks without compromising privacy. Future research will focus on extending this approach to long-form documents and addressing label imbalances in complex datasets.

Summary updated

Abstract

Due to the potential leakage of sensitive information in text, there is a societal call for feeding privacy-preserving text to model training. Recently, a lot of work showed that using synthetic text with differential privacy, rather than private text, can provide a strong privacy protection. However, achieving higher semantic similarity between synthetic and private text has not been thoroughly investigated. In this paper, we propose an approach that combines the iteratively optimized mindset from genetic algorithms to align the distribution of synthetic text with that of private text. Furthermore, not only does the final synthetic text meet the requirements of privacy protection, but also has a high level of quality. Through comparisons with various baselines on different datasets, we demonstrate that our synthetic text can closely match the utility of private text, while providing privacy protection standards robust enough to resist membership inference attacks from malicious users.
BibTeX
@inproceedings{zhao2024generate,
  author = {Wenhao Zhao and Shaoyang Song and Chunlai Zhou},
  title = {{Generate Synthetic Text Approximating the Private Distribution with Differential Privacy}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=tdPlb3iwCK},
}
Figure