FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Models

Jayneel Vora, Nader Bouacida, Aditya Krishnan, Prasant Mohapatra

TL;DR

FedDM is a new framework that brings the power of diffusion models to Federated Learning (FL), allowing multiple parties to train high-quality generative models without sharing their private data. By introducing proximal terms to handle data skew and quantization to slash communication costs by up to 4x, the authors provide a mathematically grounded and practical roadmap for collaborative generative AI.

What Problem Does This Paper Address?

Diffusion models (DMs) have become the gold standard for image generation, but they are notoriously data-hungry. Most organizations lack the massive, diverse datasets required to train these models from scratch. While Federated Learning offers a way to pool data "virtually" by training models locally and aggregating updates, applying it to DMs presents two major hurdles:

  1. Communication Overhead: Diffusion models are massive. Sending full model updates back and forth between clients and a central server creates a massive bottleneck in network bandwidth.
  2. Data Heterogeneity (Non-IID Data): In the real world, different clients have different data (e.g., one hospital has X-rays of lungs, another has X-rays of hearts). Standard federated averaging often fails or diverges when local data distributions are significantly different.

Prior research in this niche was largely limited to low-resolution images and lacked a rigorous theoretical foundation for why these models should even converge in a federated setting.

Methods

The authors propose FedDM, a suite of three algorithms built on the U-Net architecture:

  • FedDM-vanilla: A baseline implementation using standard Federated Averaging (FedAvg).
  • FedDM-prox: Designed for "label-skewed" or heterogeneous data. It adds a proximal term to the local objective function. Think of this as a "leash" that prevents local models from straying too far from the global model's parameters, which stabilizes training when a client's local data is very niche.
  • FedDM-quant: To solve the communication bottleneck, this version uses Post-Training Quantization (PTQ). It compresses 32-bit model weights into 8-bit or 16-bit integers before transmission. Crucially, it includes a calibration step where the client samples images to adjust quantization scales, minimizing the precision loss.

The paper also provides a formal Convergence Analysis using the Banach Fixed-Point Theorem, proving that the global denoising function remains a "contraction mapping"—essentially guaranteeing that the model will eventually settle on a stable solution despite being trained in pieces.

Datasets

The researchers evaluated FedDM across a range of resolutions and complexities:

  • FashionMNIST (28x28): Grayscale fashion items.
  • CIFAR-10 (32x32): Standard object categories.
  • CelebA (64x64): Celebrity faces.
  • LSUN Church Outdoors (256x256): High-resolution architectural scenes, used specifically to test Latent Diffusion Models (LDMs), which operate in a compressed latent space rather than raw pixel space.

Key Insights and Ideas

  • The "Prox" Power: The most striking insight is how much the proximal term helps. In highly skewed (non-IID) scenarios, FedDM-vanilla's performance degrades sharply (higher FID scores), while FedDM-prox remains remarkably stable.
  • Quantization is a "Free Lunch" (Almost): You can reduce the data transferred by 4x (moving from 32-bit to 8-bit) with only a marginal hit to image quality. For many practical applications, the massive speedup in training time is well worth the slight increase in Fréchet Inception Distance (FID).
  • LDM Compatibility: The framework isn't just for basic models; it scales to Latent Diffusion Models. This is critical because LDMs are the backbone of modern tools like Stable Diffusion, proving that federated training is viable for professional-grade, high-res image generation.

Key Results

  • Efficiency: FedDM-quant achieved up to a 4x reduction in total bytes transferred. For example, on CIFAR-10, 8-bit quantization reduced the data load from ~6550 MB to ~3275 MB while maintaining a respectable FID.
  • Quality: On CIFAR-10, the centralized model (trained on all data at once) achieved an FID of 5.38. FedDM-vanilla reached 6.31—a very close second that proves the federated approach doesn't sacrifice much "creativity."
  • Robustness: In non-IID tests on the CelebA dataset, FedDM-prox significantly outperformed the vanilla version, keeping the FID score low even when data was heavily skewed among clients.

Limitations and Future Work

  • Privacy Analysis: While FL is inherently more private than centralizing data, the authors acknowledge they haven't yet performed formal privacy stress tests (like gradient inversion attacks) specifically on the diffusion architecture.
  • Modality Expansion: The current study focuses exclusively on images. The authors suggest that future work should explore how FedDM handles audio, video, or text data.
  • Conditional Generation: The paper focuses on unconditional generation (creating "a church" or "a face"). Moving toward conditional models (e.g., "a church in the style of Van Gogh") in a federated setting remains an open challenge.

Summary updated

Abstract

We introduce FedDM, a novel training framework designed for the federated training of diffusion models. Our theoretical analysis establishes the convergence of diffusion models when trained in a federated setting, presenting the specific conditions under which this convergence is guaranteed. We propose a suite of training algorithms that leverage the U-Net architecture as the backbone for our diffusion models. These include a basic Federated Averaging variant, FedDM-vanilla, FedDM-prox to handle data heterogeneity among clients, and FedDM-quant, which incorporates a quantization module to reduce the model update size, thereby enhancing communication efficiency across the federated network. We evaluate our algorithms on FashionMNIST (28x28 resolution), CIFAR-10 (32x32 resolution), and CelebA (64x64 resolution) for DDPMs, as well as LSUN Church Outdoors (256x256 resolution) for LDMs, focusing exclusively on the imaging modality. Our evaluation results demonstrate that FedDM algorithms maintain high generation quality across image resolutions. At the same time, the use of quantized updates and proximal terms in the local training objective significantly enhances communication efficiency (up to 4x) and model convergence, particularly in non-IID data settings, at the cost of increased FID scores (up to 1.75x).
BibTeX
@misc{vora2024feddm,
  author = {Jayneel Vora and Nader Bouacida and Aditya Krishnan and Prasant Mohapatra},
  title = {{FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Models}},
  year = {2024},
  eprint = {2407.14730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.14730},
}
Figure