TL;DR
This paper introduces a unified benchmark to evaluate differentially private (DP) synthetic text generation across five specialized domains like healthcare, finance, and law. The authors reveal that while current state-of-the-art DP generators perform well on simple public datasets, their performance severely degrades under realistic privacy constraints when applied to complex, domain-specific text.
What Problem Does This Paper Address?
Generative AI has massive potential to unlock data sharing in high-stakes fields like medicine and finance by creating realistic synthetic datasets. To ensure this data doesn't leak sensitive information, researchers use Differential Privacy (DP)—the gold standard for mathematically bounding privacy risks.
However, existing DP text generation methods have primarily been validated on simple, open-domain datasets (like product reviews). The authors identify two major flaws with this status quo:
- Prior Exposure: Public datasets are likely already inside the pre-training corpora of large language models (LLMs). This means the models might just be memorizing and regurgitating data they've already seen, which masks actual privacy leakage and overestimates performance.
- Representativeness: General text benchmarks lack the complex jargon, long contexts, and specific organizational practices found in real-world specialized data.
To bridge this gap, this paper evaluates how DP text generation actually holds up when applied to domain-specific, gated-access datasets.
Methods
The authors establish a rigorous evaluation protocol that splits performance into two categories:
- Utility: How useful is the synthetic data for training downstream models? They measure this by training multiple classification models on the synthetic data and testing them on real, held-out data.
- Fidelity: How similar is the synthetic text to the original data? They use a suite of metrics covering surface-level similarity (BLEU, METEOR), semantic alignment (BERTScore, Universal Sentence Encoder), and corpus-level distribution (MAUVE, Named Entity Recognition overlap, and text length).
They benchmark two prominent state-of-the-art DP generation paradigms:
- DP-SGD Fine-Tuning (DP-Gen): A method that clips gradients and adds noise during the LLM fine-tuning process.
- DP Inference / Distribution Alignment (AUG-PE): A method that evolves a random dataset toward the private data distribution by privatizing a fitness function based on embedding histograms.
Datasets
The benchmark evaluates five distinct domain-specific datasets, notably including gated-access data that requires Data Usage Agreements (DUAs) to ensure the models had no prior exposure:
- HoC (Healthcare): Cancer hallmark identification in scientific literature (Open).
- N2C2’08 (Clinical): Obesity and co-morbidity recognition in clinical discharge summaries (Gated).
- PsyTAR (Healthcare): Adverse drug effect detection from social media posts (Gated).
- DMSAFN (Finance): Financial news sentiment analysis (Open).
- AsyLax (Legal): Legal reasoning and refugee case analysis (Open).
Key Insights and Ideas
- The "Prior Exposure" Illusion: LLMs perform significantly worse on gated-access datasets (like N2C2'08) than on open datasets. This strongly supports the authors' hypothesis that previous benchmarks overestimated DP text generation quality because the models had already memorized the "private" data during public pre-training.
- The Fidelity-Utility Disconnect: Different DP methods fail in different ways. DP-SGD (DP-Gen) maintains the overall structural properties (like text length) but struggles with specific domain details under strict privacy. Conversely, AUG-PE preserves statistical distributions but completely misses the nuanced tone and context of the domain.
- A Flat Utility Curve: Under strict privacy budgets ($\epsilon \le 4$), the average performance of downstream models drops to roughly 50% of real-data performance, and shockingly, stays relatively flat regardless of how much more noise is added.
Key Results
- Baseline Deficit: Even without any privacy constraints ($\epsilon = \infty$), synthetic data fails to fully match real-data baselines, showing that current generation methods struggle with domain-specific complexities.
- Severe Performance Degradation: As privacy guarantees strengthen (lower $\epsilon$), text quality drops sharply.
- Fidelity Collapse: Corpus-level fidelity metrics like MAUVE drop close to zero across all domains when DP is introduced, and entity overlap (NER) diverges heavily, particularly for long-context and gated datasets.
Limitations and Future Work
The authors acknowledge that current DP text generation is bottlenecked by foundation models pre-trained primarily on general-domain data.
To advance the field, they outline several future directions:
- Multimodal Generation: Expanding the benchmark to evaluate complex relationships across multiple data types, such as paired clinical text reports and medical images.
- Advanced Corpus Evaluation: Integrating discourse structure analysis and human assessments into the evaluation loop.
- Empirical Privacy Auditing: Developing stronger membership inference attacks (MIAs) and diagnostic datasets to empirically audit and validate the theoretical privacy guarantees of these models.
Summary updated