Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models

Matías Mendieta, Guangyu Sun, Chen Chen

TL;DR

The authors introduce FedDiff, a novel one-shot federated learning (FL) framework that leverages the high-fidelity generation of Diffusion Models to overcome the "data heterogeneity" problem. By using a new Fourier Magnitude Filtering (FMF) technique, they achieve state-of-the-art accuracy even under strict Differential Privacy (DP) constraints, outperforming existing methods by up to 20%.

What Problem Does This Paper Address?

Standard Federated Learning requires many rounds of communication between clients and a central server, which is slow and vulnerable to eavesdropping. One-shot FL solves this by requiring only a single communication round. However, it faces a massive hurdle: data heterogeneity.

When clients have very different data distributions (Non-IID data), their locally trained models become heavily biased. Reconciling these biased models into a single global model in just one shot is notoriously difficult. Previous attempts used Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) to synthesize data at the server, but these often suffer from poor sample quality and diversity—especially when noise is added to protect user privacy via Differential Privacy (DP).

Methods

The researchers propose FedDiff, which shifts the generative paradigm from VAEs/GANs to Diffusion Models (DMs).

  1. Local Training: Each client trains a class-conditioned Diffusion Model on its private data. To ensure privacy, they use DP-SGD, which involves clipping gradients and adding random noise during training.
  2. Model Aggregation: Clients send their generative models to the server.
  3. Data Synthesis & Global Training: The server uses these models to generate a massive synthetic dataset. Because this happens offline at the server, the typical "slow sampling" drawback of Diffusion Models is irrelevant.
  4. Fourier Magnitude Filtering (FMF): Under strict privacy budgets, Diffusion Models can produce "blurry" or low-quality images. The authors propose FMF, where clients send a privacy-preserved average of their data's Fourier magnitudes. The server then uses this as a "style signature" to filter out generated samples that don't match the structural characteristics of the original data.

Datasets

The approach was evaluated on three benchmarks representing different levels of complexity:

  • FashionMNIST: Grayscale images of clothing.
  • PathMNIST: A medical dataset of colon pathology images (RGB).
  • CIFAR-10: A standard natural image dataset. The authors used a Dirichlet distribution to simulate varying degrees of data heterogeneity across clients.

Key Insights and Ideas

  • Solving the Generative Trilemma: Generative models usually trade off between sample quality, diversity, and speed. While Diffusion Models are slow to sample, the authors realized that in one-shot FL, sampling speed doesn't matter because it happens once at the server. This makes DMs the perfect tool for this specific niche.
  • Simplicity is Robustness: Diffusion Models have a simpler training objective (denoising) compared to the unstable adversarial training of GANs. This simplicity makes them much more resilient to the noise introduced by Differential Privacy.
  • The Power of the Fourier Domain: FMF provides a "pragmatic" way to curate synthetic data. By looking at magnitude in the frequency domain, the server can identify which generated images have the correct "texture" and "sharpness" without ever seeing the original private images.

Key Results

FedDiff consistently outperformed all baselines (FedAvg, DENSE, FedCVAE) across all datasets:

  • Accuracy Boost: In high-heterogeneity scenarios on CIFAR-10, FedDiff achieved ~57% accuracy, while the next best generative method (FedCVAE) trailed at ~34%.
  • Privacy Resilience: Under strict Differential Privacy ($\epsilon=10$), FedDiff maintained a significant lead. For FashionMNIST, it reached ~73% accuracy, whereas other one-shot methods struggled to break 45%.
  • FMF Effectiveness: The Fourier filtering method provided an additional 2% to 4% accuracy improvement in the most challenging, high-noise DP settings.

Limitations and Future Work

While FedDiff is a major step forward, the authors note that on complex datasets like CIFAR-10 under very strict privacy budgets, the generated data still loses substantial structure.

Future directions include: * Investigating how to focus the Diffusion Models on "task-relevant" features to ignore irrelevant noise. * Exploring more advanced Differential Privacy algorithms specifically optimized for the denoising steps of Diffusion Models. * Scaling the approach to even more complex, high-resolution datasets.

Summary updated

Abstract

Federated learning (FL) enables multiple clients to train models collectively while preserving data privacy. However, FL faces challenges in terms of communication cost and data heterogeneity. One-shot federated learning has emerged as a solution by reducing communication rounds, improving efficiency, and providing better security against eavesdropping attacks. Nevertheless, data heterogeneity remains a significant challenge, impacting performance. This work explores the effectiveness of diffusion models in oneshot FL, demonstrating their applicability in addressing data heterogeneity and improving FL performance. Additionally, we investigate the utility of our diffusion model approach, FedDiff, compared to other one-shot FL methods under differential privacy (DP). Furthermore, to improve generated sample quality under DP settings, we propose a pragmatic Fourier Magnitude Filtering (FMF) method, enhancing the effectiveness of the generated data for global model training. Code available at https://github.com/mmendiet/FedDiff.
BibTeX
@inproceedings{mendieta2025navigating,
  author = {Matías Mendieta and Guangyu Sun and Chen Chen},
  title = {{Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00258},
}

Loading related content...

Figure