DP-DocLDM: Differentially Private Document Image Generation Using Latent Diffusion Models

Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract

Training high-performance document classifiers often requires access to sensitive datasets containing private information, such as medical records or financial invoices. Standard Differentially Private (DP) training methods often suffer from a significant "utility gap," where model accuracy drops sharply due to the noise required for privacy. This paper introduces DP-DocLDM, the first framework to apply DP Latent Diffusion Models to document image generation. By generating high-quality synthetic datasets under strict privacy guarantees, DP-DocLDM enables the training of downstream classifiers that achieve state-of-the-art performance without exposing sensitive real-world data.

Introduction

In the era of automated business processing, deep learning models are frequently trained on documents containing sensitive personal and corporate information. However, these models are susceptible to "memorization," where training data can be extracted through gradient inversion or membership inference attacks. Differential Privacy (DP) provides a mathematical framework to mitigate these risks by adding noise during training, ensuring that no individual record can be identified.

Despite its theoretical strengths, applying DP directly to complex image classification tasks (e.g., via DP-SGD) typically results in a massive loss of accuracy. DP-DocLDM addresses this by adopting a "generate-then-train" paradigm. Instead of training a private classifier on real data, the framework trains a private generative model to produce a synthetic version of the dataset. This synthetic data can then be used to train standard, non-private classifiers, effectively bypassing the utility gap while maintaining rigorous privacy guarantees.

Methodology

DP-DocLDM leverages Latent Diffusion Models (LDMs), which operate in a compressed latent space rather than on raw pixels. This approach is more computationally efficient and allows the model to focus on high-level semantic features, which are more resilient to the noise introduced during DP training.

The Three-Stage Pipeline

  1. Non-private Pretraining: The model is initially trained on the IIT-CDIP Test Collection, a massive public dataset of 11 million document images. This stage allows the model to learn the general structural and visual characteristics of documents without using sensitive data.
  2. Conditioning: To ensure the model generates specific document types with realistic structures, the authors employ two guidance mechanisms:
  3. Class-conditioning: Directs the model to generate specific categories (e.g., "memo" vs. "invoice").
  4. Layout-conditioning: Uses OCR-based binary masks to provide a structural "skeleton," guiding the model on where text and elements should appear.
  5. Differentially Private Fine-tuning: The pretrained model is fine-tuned on the private target dataset using DP-specific algorithms, such as DPDM (which utilizes noise multiplicity to improve gradient estimation) and DP-Promise.

Per-Label Private Fine-tuning

A significant technical innovation in DP-DocLDM is the use of per-label private fine-tuning. Training a single model to distinguish between many classes under heavy DP noise is often ineffective. By training independent private models for each document category, the authors significantly improved the quality and class-specificity of the generated images.

Experimental Results

The framework was evaluated on two primary benchmarks: RVL-CDIP (400,000 images across 16 categories) and Tobacco3482 (a smaller, imbalanced dataset of 3,482 images).

Performance on Small-Scale Data

The most dramatic results were observed on the Tobacco3482 dataset. A standard ResNet-50 trained directly with DP-SGD achieved only 44-46% accuracy. In contrast, the same ResNet-50 trained on synthetic data generated by DP-DocLDM reached approximately 78% accuracy. This demonstrates that generative DP methods can effectively overcome data scarcity issues that plague direct DP training.

Benchmarking Large Datasets

On the larger RVL-CDIP dataset, the approach achieved an accuracy of 80.21% at a privacy budget of $\epsilon=1$. This performance is competitive with state-of-the-art direct private training methods while providing the added benefit of a reusable, private synthetic dataset.

Key Insights

  • Structural Guidance: Layout conditioning via OCR masks is critical. It provides a structural hint that helps the model maintain document-like formatting even when high levels of DP noise are applied.
  • Latent Space Efficiency: By using an autoencoder to move into latent space (specifically using a KL-F4 downscaling factor), the model avoids the difficulties of learning high-frequency pixel noise under DP constraints.
  • Privacy-Utility Trade-off: The "generate-then-train" approach allows for the creation of high-fidelity synthetic documents that preserve the visual characteristics of the original classes (e.g., scientific reports vs. handwritten notes) even at strict privacy levels.

Limitations and Future Work

While DP-DocLDM represents a significant step forward, it involves a multi-stage pipeline that is more computationally intensive than standard training. Additionally, DP training remains sensitive to hyperparameters, such as noise multipliers and clipping thresholds. Future research will focus on extending this generative approach to more complex document tasks, including layout analysis, handwriting recognition, and table extraction, as well as exploring advanced sampling techniques to further improve image diversity.

Summary updated

BibTeX
@inproceedings{saifullah2025dp,
  author = {Saifullah Saifullah and Stefan Agne and Andreas Dengel and Sheraz Ahmed},
  title = {{DP-DocLDM: Differentially Private Document Image Generation Using Latent Diffusion Models}},
  booktitle = {ICDAR},
  year = {2025},
  doi = {10.1007/978-3-032-04627-7\_6},
}
Figure