Evaluating Differentially Private Generation of Domain-Specific Text

Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar, Iqra Zahid, Yuping Wu, Warren Del-Pinto, Goran Nenadic, Siew-Kei Lam, Jie Zhang, Anil A Bharath

TL;DR

This paper introduces a unified benchmark to evaluate differentially private (DP) synthetic text generation across five specialized domains like healthcare, finance, and law. The authors reveal that while current state-of-the-art DP generators perform well on simple public datasets, their performance severely degrades under realistic privacy constraints when applied to complex, domain-specific text.

What Problem Does This Paper Address?

Generative AI has massive potential to unlock data sharing in high-stakes fields like medicine and finance by creating realistic synthetic datasets. To ensure this data doesn't leak sensitive information, researchers use Differential Privacy (DP)—the gold standard for mathematically bounding privacy risks.

However, existing DP text generation methods have primarily been validated on simple, open-domain datasets (like product reviews). The authors identify two major flaws with this status quo:

  1. Prior Exposure: Public datasets are likely already inside the pre-training corpora of large language models (LLMs). This means the models might just be memorizing and regurgitating data they've already seen, which masks actual privacy leakage and overestimates performance.
  2. Representativeness: General text benchmarks lack the complex jargon, long contexts, and specific organizational practices found in real-world specialized data.

To bridge this gap, this paper evaluates how DP text generation actually holds up when applied to domain-specific, gated-access datasets.

Methods

The authors establish a rigorous evaluation protocol that splits performance into two categories:

  • Utility: How useful is the synthetic data for training downstream models? They measure this by training multiple classification models on the synthetic data and testing them on real, held-out data.
  • Fidelity: How similar is the synthetic text to the original data? They use a suite of metrics covering surface-level similarity (BLEU, METEOR), semantic alignment (BERTScore, Universal Sentence Encoder), and corpus-level distribution (MAUVE, Named Entity Recognition overlap, and text length).

They benchmark two prominent state-of-the-art DP generation paradigms:

  1. DP-SGD Fine-Tuning (DP-Gen): A method that clips gradients and adds noise during the LLM fine-tuning process.
  2. DP Inference / Distribution Alignment (AUG-PE): A method that evolves a random dataset toward the private data distribution by privatizing a fitness function based on embedding histograms.
Figure 1: We benchmark differentially private synthetic text generators using a variety of metrics and find that they strug- gle to produce realistic data from specialist domains.

Datasets

The benchmark evaluates five distinct domain-specific datasets, notably including gated-access data that requires Data Usage Agreements (DUAs) to ensure the models had no prior exposure:

  • HoC (Healthcare): Cancer hallmark identification in scientific literature (Open).
  • N2C2’08 (Clinical): Obesity and co-morbidity recognition in clinical discharge summaries (Gated).
  • PsyTAR (Healthcare): Adverse drug effect detection from social media posts (Gated).
  • DMSAFN (Finance): Financial news sentiment analysis (Open).
  • AsyLax (Legal): Legal reasoning and refugee case analysis (Open).
Table 1: Benchmark characteristics: Number of labels (|𝐶|), dataset size (|𝐷|), average/90th percentile token length |𝐿| and data access.

Key Insights and Ideas

  • The "Prior Exposure" Illusion: LLMs perform significantly worse on gated-access datasets (like N2C2'08) than on open datasets. This strongly supports the authors' hypothesis that previous benchmarks overestimated DP text generation quality because the models had already memorized the "private" data during public pre-training.
  • The Fidelity-Utility Disconnect: Different DP methods fail in different ways. DP-SGD (DP-Gen) maintains the overall structural properties (like text length) but struggles with specific domain details under strict privacy. Conversely, AUG-PE preserves statistical distributions but completely misses the nuanced tone and context of the domain.
  • A Flat Utility Curve: Under strict privacy budgets ($\epsilon \le 4$), the average performance of downstream models drops to roughly 50% of real-data performance, and shockingly, stays relatively flat regardless of how much more noise is added.

Key Results

  • Baseline Deficit: Even without any privacy constraints ($\epsilon = \infty$), synthetic data fails to fully match real-data baselines, showing that current generation methods struggle with domain-specific complexities.
  • Severe Performance Degradation: As privacy guarantees strengthen (lower $\epsilon$), text quality drops sharply.
Downstream classification F1 scores across different privacy budgets ($\epsilon$).
  • Fidelity Collapse: Corpus-level fidelity metrics like MAUVE drop close to zero across all domains when DP is introduced, and entity overlap (NER) diverges heavily, particularly for long-context and gated datasets.
Figure 2: Results for “many reference” metrics of DP-Gen and AUG-PE generated data at different 𝜖levels, expressed as relative decrease compared to non-privately synthesised data. As noise increases, the quality deteriorates.

Limitations and Future Work

The authors acknowledge that current DP text generation is bottlenecked by foundation models pre-trained primarily on general-domain data.

To advance the field, they outline several future directions:

  • Multimodal Generation: Expanding the benchmark to evaluate complex relationships across multiple data types, such as paired clinical text reports and medical images.
  • Advanced Corpus Evaluation: Integrating discourse structure analysis and human assessments into the evaluation loop.
  • Empirical Privacy Auditing: Developing stronger membership inference attacks (MIAs) and diagnostic datasets to empirically audit and validate the theoretical privacy guarantees of these models.

Summary updated

Abstract

Generative AI offers transformative potential for high-stakes domains such as healthcare and finance, yet privacy and regulatory barriers hinder the use of real-world data. To address this, differentially private synthetic data generation has emerged as a promising alternative. In this work, we introduce a unified benchmark to systematically evaluate the utility and fidelity of text datasets generated under formal Differential Privacy (DP) guarantees. Our benchmark addresses key challenges in domain-specific benchmarking, including choice of representative data and realistic privacy budgets, accounting for pre-training and a variety of evaluation metrics. We assess state-of-the-art privacy-preserving generation methods across five domain-specific datasets, revealing significant utility and fidelity degradation compared to real data, especially under strict privacy constraints. These findings underscore the limitations of current approaches, outline the need for advanced privacy-preserving data sharing methods and set a precedent regarding their evaluation in realistic scenarios.
BibTeX
@misc{sun2025evaluating,
  author = {Yidan Sun and Viktor Schlegel and Srinivasan Nandakumar and Iqra Zahid and Yuping Wu and Warren Del-Pinto and Goran Nenadic and Siew-Kei Lam and Jie Zhang and Anil A Bharath},
  title = {{Evaluating Differentially Private Generation of Domain-Specific Text}},
  year = {2025},
  eprint = {2508.20452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.20452},
}
Figure