TL;DR
Synthesis via Private Textual Intermediaries (SPTI) bypasses the massive computational cost of training differentially private (DP) image generators by shifting the privacy-preserving process from pixels to text. By leveraging off-the-shelf foundation models via APIs, SPTI generates high-resolution, privacy-preserving synthetic images with state-of-the-art fidelity and diversity without requiring any model training.
What Problem Does This Paper Address?
Sharing sensitive visual data—such as medical imagery or user interface screenshots—for research and analysis is heavily restricted due to privacy concerns. Differentially private (DP) synthetic data generation offers a solution by creating a mathematical guarantee that individual data points cannot be leaked.
However, existing approaches face a steep trade-off:
- DP Fine-tuning (e.g., DP-SGD): Requires adjusting millions of parameters under noisy constraints. This demands massive computational resources (hundreds of GPUs) and is completely incompatible with powerful, closed-source proprietary models available only via APIs.
- Image-space Private Evolution: Generates data using inference-only APIs but struggles to maintain structural integrity and fine-grained details when scaling up to high-resolution images.
Directly adding DP noise to high-dimensional pixel spaces destroys image quality. The authors address this by asking: Can we use natural language as a lower-dimensional, semantically rich bridge to synthesize high-resolution private images?
Methods
SPTI introduces a three-stage, training-free framework that operates entirely through model inference:
- Images to Texts: A standard captioning model translates the private image dataset into a corpus of concise text descriptions.
- Private Evolution in Text Space: The core privacy mechanism uses a modified version of Augmented Private Evolution (Aug-PE). Instead of voting on text similarity, SPTI introduces Image Voting. Candidate text descriptions are rendered into images using a text-to-image model, and these generated images are compared to the private images within an embedding space (like Inception-v3). The closest matches receive votes, DP noise is added to the histogram, and the top text candidates are selected and mutated for the next generation.
- Evolved Texts to Images: The final DP-sanitized captions are fed into a high-quality text-to-image diffusion model to produce the final high-resolution synthetic dataset. Because the privacy-critical selection happened upstream in the text space, the final images are inherently DP-compliant via the post-processing immunity property of differential privacy.
Datasets
The authors evaluated SPTI across six diverse and privacy-sensitive datasets:
- LSUN Bedroom: Captures private living spaces.
- Cat: A standard image synthesis benchmark.
- European Art: Niche artistic styles.
- Wave-ui-25k: User interface screenshots containing potentially sensitive user data.
- MM-CelebA-HQ & CelebA: High-resolution human faces (used for downstream classification tasks).
- Sprite Fright (New): A novel dataset constructed by the authors by extracting frames from the open-source Blender movie Sprite Fright. This was specifically introduced to mitigate concerns about data contamination in the pre-trained foundation models.
Key Insights and Ideas
- Text as a Universal Interface: Shifting the DP bottleneck from pixels to text leverages the fact that large language models are exceptionally robust to private text generation. Text acts as a natural dimensionality reduction technique while preserving core semantics.
- The Disconnect of Text-Only Voting: A key "aha" moment was realizing that text-space similarity does not guarantee image-space similarity. Because text-to-image models are complex and stochastic, two similar captions can yield wildly different images. Image Voting fixes this by evaluating the text candidates based on their actual visual outputs against the private images.
- Proprietary Model Compatibility: Because SPTI is entirely inference-based, researchers can plug-and-play the strongest commercial models (like GPT-4o or DALL-E 3) without needing access to raw weights or gradient clipping hooks.
Key Results
SPTI achieved substantial quantitative improvements in Fréchet Inception Distance (FID) scores under strict privacy guarantees ($\epsilon = 1.0$, where lower FID indicates better quality):
- LSUN Bedroom: SPTI achieved an FID of 26.71, significantly outperforming pixel-space Private Evolution (40.36) and DP fine-tuning (31.76).
- MM-CelebA-HQ: SPTI reached an FID of 33.27, a massive leap over the DP fine-tuning baseline of 57.01.
In downstream tasks, a classifier trained on CelebA faces synthesized by SPTI achieved much higher accuracy than those trained on standard Private Evolution data, proving that SPTI preserves the diversity and utility required for real-world machine learning applications.
Limitations and Future Work
- Domain Generalization: SPTI is fundamentally bound by the coverage of its underlying foundation models. If the pre-trained models have never seen specialized data (e.g., rare medical conditions or satellite imagery), the framework will fail to generate accurate representations.
- Computational Overhead: While it avoids expensive training, the iterative generation process (LLM text mutation + diffusion rendering for voting) is slow. A full run takes roughly 15 to 19 hours on modern hardware, making pipeline optimization (like prompt caching or model distillation) a priority for future work.
Summary updated