ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning

Kaan Ozkara, Bruce Huang, Ruida Zhou, Suhas Diggavi

Abstract

Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to local data statistics. Though there has been a plethora of algorithms proposed for personalized supervised learning, discovering the structure of local data through personalized unsupervised learning is less explored. We initiate a systematic study of such personalized unsupervised learning by developing algorithms based on optimization criteria inspired by a hierarchical Bayesian statistical framework. We develop adaptive algorithms that discover the balance between using limited local data and collaborative information. We do this in the context of two unsupervised learning tasks: personalized dimensionality reduction (ADEPT-PCA and ADEPT-AE) and personalized diffusion models (ADEPT-DGM). We develop convergence analyses for our adaptive algorithms which illustrate the dependence on problem parameters (e.g., heterogeneity, local sample size). We also develop a theoretical framework for personalized diffusion models, which shows the benefits of collaboration even under heterogeneity. We finally evaluate our proposed algorithms using synthetic and real data, demonstrating the effective sample amplification for personalized tasks, induced through collaboration, despite data heterogeneity.

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized devices while keeping data localized. While significant progress has been made in personalized supervised learning, the unsupervised domain—focused on discovering underlying data structures—remains relatively unexplored. Researchers from UCLA have introduced ADEPT (Adaptive Distributed Empirical-Bayes based Personalized Training) to bridge this gap.

The fundamental challenge in this space is statistical heterogeneity. In a federated network, each client (e.g., a mobile phone or a hospital) possesses a unique data distribution. A single "global" model often fails to capture these nuances, yet individual clients rarely have enough local data to train high-quality models independently. ADEPT addresses this by adaptively balancing local data with collaborative global information, enabling complex tasks like dimensionality reduction and image generation even with tiny, diverse datasets.

Methodology: Hierarchical Bayesian Framework

The key innovation of ADEPT is its formulation of the federated system as a Hierarchical Bayesian model. This approach provides a mathematically rigorous way to handle the tension between global collaboration and local personalization.

The hierarchical Bayesian framework used in ADEPT, where population parameters $\Gamma$ serve as a prior for local client parameters $\theta_i$.

As shown in the diagram above, the model consists of:

  1. Local Models ($\theta_i$): Parameters specifically tailored to each client's local data.
  2. Global Population Model ($\Gamma$): A shared distribution that acts as a "prior" for the local models, capturing commonalities across the entire population.
  3. Adaptive Balancing: Unlike traditional methods that use fixed weights for collaboration, ADEPT adaptively estimates the discrepancy ($\sigma$) between global and local information. This variance parameter acts as an automated "trust" knob; if a client's data deviates significantly from the group, $\sigma$ increases, allowing the client to prioritize its local data and avoid being overwhelmed by an irrelevant global average.

The authors apply this framework to three major tasks: ADEPT-PCA for linear dimensionality reduction, ADEPT-AE for non-linear autoencoders, and ADEPT-DGM for personalized diffusion generative models. For diffusion models, where likelihood functions are often intractable, the authors utilize the Evidence Lower Bound (ELBO) as a surrogate to derive optimization criteria.

Experimental Setup and Datasets

The researchers evaluated ADEPT using a combination of synthetic benchmarks and standard computer vision datasets:

  • Synthetic Data: Used to validate theoretical convergence and analyze the impact of varying noise and heterogeneity levels.
  • MNIST & Fashion MNIST: Distributed "pathologically" (where each client sees only one or two classes) to simulate extreme heterogeneity.
  • CIFAR-10: Employed for more complex autoencoder and diffusion model tasks.

Key Insights and Results

The experimental results highlight several critical advantages of the ADEPT framework:

Sample Amplification and Equity

One of the most striking findings is the "sample amplification" effect. On the CIFAR-10 dataset, ADEPT-AE achieved performance with only 250 local samples that matched a local training baseline using 5,000 samples—effectively a 20x amplification of local data through collaboration.

Furthermore, ADEPT demonstrates a significant "equity" effect. In diffusion models, the framework improves the performance of the "worst-off" clients. By reducing the variance in performance across the network, ADEPT ensures a fairer distribution of model quality. This is supported by theoretical guarantees (Theorem 4.4) showing that collaboration via ADEPT improves generated sample quality over purely local training, even under heterogeneity.

Quantitative Performance

ADEPT consistently outperformed standard FedAvg and local training baselines. In dimensionality reduction tasks, ADEPT-AE reduced reconstruction error on MNIST by approximately 25-35% compared to baselines. In generative modeling, ADEPT-DGM achieved superior results in terms of Kernel Inception Distance (KID).

Quantitative comparison of Kernel Inception Distance (KID) scores for personalized diffusion models. Lower values indicate higher generative quality.

As seen in the table above, ADEPT-DGM achieved a KID of 0.069 on MNIST, significantly outperforming FedAvg with fine-tuning (0.090) and purely local training (0.083). Qualitatively, the images produced by ADEPT-DGM exhibit significantly less background noise and fewer "hallucinations" compared to standard federated approaches.

Conclusion and Future Directions

ADEPT provides a robust framework for personalized federated unsupervised learning, yet several areas for future work remain. The authors acknowledge potential privacy risks, noting that while raw data is not shared, malicious actors could potentially infer distribution information from the global model. Integrating formal privacy guarantees, such as Differential Privacy, is a key next step. Additionally, extending the theoretical analysis beyond Gaussian distributions and improving the communication efficiency of these algorithms for real-world edge computing are important future goals.

Summary updated

Abstract

Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to local data statistics. Though there has been a plethora of algorithms proposed for personalized supervised learning, discovering the structure of local data through personalized unsupervised learning is less explored. We initiate a systematic study of such personalized unsupervised learning by developing algorithms based on optimization criteria inspired by a hierarchical Bayesian statistical framework. We develop adaptive algorithms that discover the balance between using limited local data and collaborative information. We do this in the context of two unsupervised learning tasks: personalized dimensionality reduction (ADEPT-PCA and ADEPT-AE) and personalized diffusion models (ADEPT-DGM). We develop convergence analyses for our adaptive algorithms which illustrate the dependence on problem parameters (e.g., heterogeneity, local sample size). We also develop a theoretical framework for personalized diffusion models, which shows the benefits of collaboration even under heterogeneity. We finally evaluate our proposed algorithms using synthetic and real data, demonstrating the effective sample amplification for personalized tasks, induced through collaboration, despite data heterogeneity.
BibTeX
@inproceedings{ozkara2025adept,
  author = {Kaan Ozkara and Bruce Huang and Ruida Zhou and Suhas Diggavi},
  title = {{ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=z9RDCfRZHw},
}
Figure