Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries

Haoxiang Wang, Zinan Lin, Da Yu, Huishuai Zhang

TL;DR

Synthesis via Private Textual Intermediaries (SPTI) bypasses the massive computational cost of training differentially private (DP) image generators by shifting the privacy-preserving process from pixels to text. By leveraging off-the-shelf foundation models via APIs, SPTI generates high-resolution, privacy-preserving synthetic images with state-of-the-art fidelity and diversity without requiring any model training.

What Problem Does This Paper Address?

Sharing sensitive visual data—such as medical imagery or user interface screenshots—for research and analysis is heavily restricted due to privacy concerns. Differentially private (DP) synthetic data generation offers a solution by creating a mathematical guarantee that individual data points cannot be leaked.

However, existing approaches face a steep trade-off:

  1. DP Fine-tuning (e.g., DP-SGD): Requires adjusting millions of parameters under noisy constraints. This demands massive computational resources (hundreds of GPUs) and is completely incompatible with powerful, closed-source proprietary models available only via APIs.
  2. Image-space Private Evolution: Generates data using inference-only APIs but struggles to maintain structural integrity and fine-grained details when scaling up to high-resolution images.

Directly adding DP noise to high-dimensional pixel spaces destroys image quality. The authors address this by asking: Can we use natural language as a lower-dimensional, semantically rich bridge to synthesize high-resolution private images?

Methods

SPTI introduces a three-stage, training-free framework that operates entirely through model inference:

  1. Images to Texts: A standard captioning model translates the private image dataset into a corpus of concise text descriptions.
  2. Private Evolution in Text Space: The core privacy mechanism uses a modified version of Augmented Private Evolution (Aug-PE). Instead of voting on text similarity, SPTI introduces Image Voting. Candidate text descriptions are rendered into images using a text-to-image model, and these generated images are compared to the private images within an embedding space (like Inception-v3). The closest matches receive votes, DP noise is added to the histogram, and the top text candidates are selected and mutated for the next generation.
  3. Evolved Texts to Images: The final DP-sanitized captions are fed into a high-quality text-to-image diffusion model to produce the final high-resolution synthetic dataset. Because the privacy-critical selection happened upstream in the text space, the final images are inherently DP-compliant via the post-processing immunity property of differential privacy.
Figure 1: Overview of the Synthesis via Private Textual Intermediaries (SPTI) framework for differentially private (DP) synthetic data generation. Top left: DP fine-tuning framework. A pretrained model is fine-tuned on private data under DP constraints, and the resulting model is used to generate DP synthetic data. Top right: Private Evolution (PE) on images. This method begins by randomly generating candidate samples, which are then compared to private data. Samples are selected based on a voting mechanism and further perturbed to produce a new generation of samples. Bottom: Synthesis via Private Textual Intermediaries (SPTI) framework. Private image data is served as reference. A modified Augmented Private Evolution (Aug-PE) method [53] is then applied to generate DP synthetic text data, which is subsequently transformed into DP synthetic images using a diffusion model API.

Datasets

The authors evaluated SPTI across six diverse and privacy-sensitive datasets:

  • LSUN Bedroom: Captures private living spaces.
  • Cat: A standard image synthesis benchmark.
  • European Art: Niche artistic styles.
  • Wave-ui-25k: User interface screenshots containing potentially sensitive user data.
  • MM-CelebA-HQ & CelebA: High-resolution human faces (used for downstream classification tasks).
  • Sprite Fright (New): A novel dataset constructed by the authors by extracting frames from the open-source Blender movie Sprite Fright. This was specifically introduced to mitigate concerns about data contamination in the pre-trained foundation models.

Key Insights and Ideas

  • Text as a Universal Interface: Shifting the DP bottleneck from pixels to text leverages the fact that large language models are exceptionally robust to private text generation. Text acts as a natural dimensionality reduction technique while preserving core semantics.
  • The Disconnect of Text-Only Voting: A key "aha" moment was realizing that text-space similarity does not guarantee image-space similarity. Because text-to-image models are complex and stochastic, two similar captions can yield wildly different images. Image Voting fixes this by evaluating the text candidates based on their actual visual outputs against the private images.
  • Proprietary Model Compatibility: Because SPTI is entirely inference-based, researchers can plug-and-play the strongest commercial models (like GPT-4o or DALL-E 3) without needing access to raw weights or gradient clipping hooks.

Key Results

SPTI achieved substantial quantitative improvements in Fréchet Inception Distance (FID) scores under strict privacy guarantees ($\epsilon = 1.0$, where lower FID indicates better quality):

  • LSUN Bedroom: SPTI achieved an FID of 26.71, significantly outperforming pixel-space Private Evolution (40.36) and DP fine-tuning (31.76).
  • MM-CelebA-HQ: SPTI reached an FID of 33.27, a massive leap over the DP fine-tuning baseline of 57.01.
Figure 5: Experiment results on LSUN Bedroom dataset

In downstream tasks, a classifier trained on CelebA faces synthesized by SPTI achieved much higher accuracy than those trained on standard Private Evolution data, proving that SPTI preserves the diversity and utility required for real-world machine learning applications.

Limitations and Future Work

  • Domain Generalization: SPTI is fundamentally bound by the coverage of its underlying foundation models. If the pre-trained models have never seen specialized data (e.g., rare medical conditions or satellite imagery), the framework will fail to generate accurate representations.
  • Computational Overhead: While it avoids expensive training, the iterative generation process (LLM text mutation + diffusion rendering for voting) is slow. A full run takes roughly 15 to 19 hours on modern hardware, making pipeline optimization (like prompt caching or model distillation) a priority for future work.

Summary updated

Abstract

Generating high fidelity, differentially private (DP) synthetic images offers a promising route to share and analyze sensitive visual data without compromising individual privacy. However, existing DP image synthesis methods struggle to produce high resolution outputs that faithfully capture the structure of the original data. In this paper, we introduce a novel method, referred to as Synthesis via Private Textual Intermediaries (SPTI), that can generate high resolution DP images with easy adoption. The key idea is to shift the challenge of DP image synthesis from the image domain to the text domain by leveraging state of the art DP text generation methods. SPTI first summarizes each private image into a concise textual description using image to text models, then applies a modified Private Evolution algorithm to generate DP text, and finally reconstructs images using text to image models. Notably, SPTI requires no model training, only inference with off the shelf models. Given a private dataset, SPTI produces synthetic images of substantially higher quality than prior DP approaches. On the LSUN Bedroom dataset, SPTI attains an FID equal to 26.71 under epsilon equal to 1.0, improving over Private Evolution FID of 40.36. Similarly, on MM CelebA HQ, SPTI achieves an FID equal to 33.27 at epsilon equal to 1.0, compared to 57.01 from DP fine tuning baselines. Overall, our results demonstrate that Synthesis via Private Textual Intermediaries provides a resource efficient and proprietary model compatible framework for generating high resolution DP synthetic images, greatly expanding access to private visual datasets.
BibTeX
@inproceedings{wang2025synthesize,
  author = {Haoxiang Wang and Zinan Lin and Da Yu and Huishuai Zhang},
  title = {{Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=g8zr9rxRHm},
}
Figure