Abstract
Federated Learning (FL) traditionally focuses on supervised tasks, leaving personalized unsupervised learning—such as dimensionality reduction and generative modeling—relatively unexplored. We present ADEPT (Adaptive Distributed Empirical-Bayes based Personalized Training), a framework that utilizes Hierarchical Bayesian modeling to bridge this gap. By treating the global model as a population prior and adaptively learning the discrepancy between local and global distributions, ADEPT enables clients to leverage collective knowledge while preserving local nuances. Our results demonstrate that ADEPT can achieve up to a 20x effective increase in local sample size, significantly outperforming standard global or purely local training methods across PCA, Auto-encoders, and Diffusion Generative Models.
Introduction
In Federated Learning, data heterogeneity remains a primary obstacle. While personalization techniques have matured for supervised classification, unsupervised tasks face unique challenges. Individual clients often possess insufficient local data to accurately model their specific data distributions, yet a single "one-size-fits-all" global model often fails to capture the diversity of local environments. This "heterogeneity trap" is particularly acute in tasks like dimensionality reduction or data generation, where the underlying structure of the data is unknown.
To address these challenges, we introduce the ADEPT framework. ADEPT formalizes personalized unsupervised learning through a Hierarchical Bayesian lens, allowing for a principled balance between local observations and global collaborative information.
The ADEPT Framework
Adaptive Hierarchical Modeling
The core innovation of ADEPT is its treatment of the global model as a "population prior." Unlike previous approaches that require a fixed, manually tuned hyperparameter to balance local and global influence, ADEPT treats the discrepancy (variance $\sigma$) as a learnable parameter. This adaptive balancing allows the system to automatically determine the degree of trust a client should place in the global prior:
- Unique Data: If a client's data is highly idiosyncratic, the algorithm learns to rely more heavily on local observations.
- Standard Data: If the data aligns well with the global trend, the algorithm leans on the collaborative model to compensate for small local sample sizes.
Applications: From PCA to Diffusion Models
ADEPT is designed to be versatile, supporting both linear and non-linear unsupervised tasks:
- ADEPT-PCA & Auto-encoders (AE): For linear dimensionality reduction, the framework solves optimization problems on the Stiefel manifold—the mathematical space of orthogonal matrices. This ensures rigorous convergence and stability during distributed training.
- ADEPT-DGM (Diffusion Generative Models): The framework extends to state-of-the-art generative modeling. By leveraging the "collective wisdom" of the network, clients can generate high-quality samples that match their specific local distribution, even when their own training set is extremely limited.
Experimental Evaluation
Methodology
We evaluated ADEPT using a combination of synthetic datasets (for controlled noise and heterogeneity) and real-world benchmarks, including MNIST, Fashion MNIST, and CIFAR-10. To simulate extreme real-world conditions, we employed a "pathological heterogeneity" setup where each client only had access to a single class of data (e.g., one client seeing only the digit '0').
Key Results
- Sample Amplification: One of the most significant findings is the "force multiplier" effect of collaboration. On the CIFAR-10 dataset, ADEPT demonstrated a 20x effective sample amplification. A client with only 250 samples achieved performance parity with a client training in isolation on 5,000 samples.
- Reconstruction and Generation Quality: In Auto-encoder tasks, ADEPT reduced reconstruction error by approximately 35% compared to local training and 25% compared to standard FedAvg. For Diffusion models, ADEPT achieved superior generative quality, reaching a Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) of 0.067 on MNIST, compared to 0.082 for standard fine-tuning.
- Equity for Outliers: The framework provides the most substantial benefits to "worst-case" clients—those with the most difficult or outlier data distributions—ensuring that the benefits of the federated network are distributed more equitably.
Discussion and Insights
ADEPT demonstrates that collaboration in FL is not just about reaching a global consensus, but about enriching local models. By learning the prior simultaneously with the local models, the system avoids the need for centralized pre-training, which is often impossible due to privacy constraints. Furthermore, our theoretical analysis provides conditions under which collaboration is guaranteed to improve performance, even in the presence of significant data diversity.
Limitations and Future Work
While ADEPT is statistically robust, several avenues for future research remain:
- Communication Efficiency: The adaptive updates require frequent communication; optimizing these for edge devices with limited bandwidth is a priority.
- Privacy: Integrating formal privacy guarantees, such as Differential Privacy, into the Hierarchical Bayes framework is an essential next step.
- Task Expansion: We aim to apply the ADEPT criterion to other unsupervised domains, such as self-supervised representation learning and clustering.
Conclusion
ADEPT provides a rigorous and practical framework for personalized unsupervised learning in federated settings. By adaptively balancing local data with a learned global prior, it overcomes the limitations of small local sample sizes and data heterogeneity, turning the challenge of diversity into a collaborative advantage.
Summary updated