Generate Synthetic Text Approximating the Private Distribution with Differential Privacy

Wenhao Zhao, S. H. Song, Chunlai Zhou, Hua Zhang, Zhong Zhou

Abstract: As Large Language Models (LLMs) become increasingly integrated into sensitive applications, the risk of leaking private training data through Membership Inference Attacks (MIA) has grown. While Differential Privacy (DP) provides a theoretical framework for protection, its direct application to LLM training is often computationally prohibitive. This paper introduces a novel iterative framework that leverages genetic algorithms to generate high-quality synthetic text embeddings that approximate a private distribution. By evolving a population of synthetic samples through noisy selection and estimation of distribution algorithms, our method produces utility-preserving demonstrations for in-context learning. Experimental evaluations on AGNews, Disaster, and Trec datasets demonstrate that our approach achieves superior accuracy and distribution alignment compared to existing baselines while maintaining robust privacy guarantees.

Introduction

The ubiquity of Large Language Models (LLMs) has brought data privacy to the forefront of machine learning research. Textual datasets often contain sensitive information—such as personal identifiers or unique linguistic patterns—that can be exploited by attackers to determine if a specific record was part of a model's training set. Traditional text sanitization methods, which rely on masking specific words, often fail to protect the underlying semantic structure of the data.

To address these challenges, we propose a method to generate synthetic text that mimics the semantic distribution of private data without exposing the raw samples. Our approach bridges the gap between data utility and privacy by providing synthetic "demonstrations" that can be used for downstream tasks like In-Context Learning (ICL) without compromising the privacy of the original data owners.

Methodology

Our framework employs an iterative approach inspired by Genetic Algorithms (GA) to align a synthetic population with a private dataset. The process is divided into several key stages:

Overview of the proposed iterative synthetic text generation framework, illustrating the flow from private/public text to the final synthetic output.

1. Embedding and Decoding

We first train a text decoder on public data to map high-dimensional embeddings back into coherent natural language. This ensures that the evolutionary process, which occurs in the embedding space, results in readable text.

2. Iterative Evolution

The core of the method is a population-based evolution:

  • Private Selection: Private samples "vote" for synthetic samples that are semantically closest using cosine distance. To maintain differential privacy, we utilize the LimitedDomain mechanism, which adds calibrated noise to these vote counts.
  • Offspring Generation: Using an Estimation of Distribution Algorithm (EDA), we calculate the mean and variance of the selected "parent" embeddings and sample new "offspring" from a Gaussian distribution. This allows the synthetic population to gradually drift toward the private distribution over multiple iterations.

3. Perplexity Filtering

To prevent the genetic algorithm from moving embeddings into regions that produce incoherent text, we employ a pre-trained GPT-2 model to check the perplexity of the decoded text. Samples that fall below a readability threshold are discarded, ensuring the final output remains linguistically valid.

Experimental Results

We evaluated our framework using In-Context Learning (ICL) across three diverse classification datasets: AGNews, Disaster, and Trec. We compared our method against several baselines, including CusText, DP-ICL, and SentDP, using various OpenAI model backbones (babbage, curie, and davinci).

Performance comparison across multiple datasets and LLM backbones, showing accuracy and standard deviation.

Utility and Accuracy

As shown in the results table, our method consistently outperforms baselines, particularly when the privacy budget is tight (e.g., $\epsilon=5$). On the AGNews dataset, our approach achieved 68.9% accuracy with the davinci model, surpassing the next best baseline by nearly 2%. This indicates that the iterative alignment process successfully captures the nuances of the private distribution better than "one-shot" generation methods.

Distribution Alignment and Privacy

Using Wasserstein distance as a metric, we observed that the synthetic population moves significantly closer to the private distribution with each iteration. Furthermore, the reliance on noisy distribution estimation rather than direct transformation makes the synthetic text highly resistant to Membership Inference Attacks (MIA). The iterative nature of the GA approach also provides more stable and consistent results across different runs compared to methods like SentDP.

Conclusion and Future Work

This research demonstrates that genetic algorithms combined with differential privacy can effectively generate synthetic text that preserves both utility and privacy. While the method shows great promise for short-to-medium length texts, future work will focus on scaling the framework to handle longer documents where information loss during embedding compression is more pronounced. Additionally, we aim to further optimize the decoder to handle more complex linguistic structures and reduce the privacy budget consumption per iteration.

Summary updated

BibTeX
@inproceedings{zhao2024generate,
  author = {Wenhao Zhao and S. H. Song and Chunlai Zhou and Hua Zhang and Zhong Zhou},
  title = {{Generate Synthetic Text Approximating the Private Distribution with Differential Privacy}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=tdPlb3iwCK},
}
Figure