Introduction
Standard Federated Learning (FL) aims to train a single global model across many decentralized devices. However, real-world data is rarely uniform; a user in Sydney might have different data patterns than a user in New York—a phenomenon known as the "Non-IID" problem. Personalized Federated Learning (PFL) attempts to address this by balancing global knowledge with local bias.
While most PFL methods focus on architectural changes or optimization constraints, this research introduces FedDVA (Federated Dual Variational Autoencoder), which shifts the focus to latent representations. The goal is to explicitly separate "universal knowledge" (e.g., the core shape of a digit) from "client-specific traits" (e.g., unique handwriting styles or watermarks).
The FedDVA Framework
FedDVA introduces a dual-encoder architecture to achieve this disentanglement. Unlike a standard VAE that uses a single encoder, FedDVA employs two distinct probabilistic encoders to process data $x$:
- Universal Encoder $f(x)$: This extracts the universal representation ($z$). It is constrained by a standard Gaussian prior to ensure it captures general features shared across the entire client population.
- Personalized Encoder $h(x, z)$: This extracts the personalized representation ($c$), conditioned on both the raw data and the universal features.
To optimize this in a federated setting, the authors utilize a client-specific Evidence Lower Bound (ELBO). They introduce a "slack regularizer" to handle the unknown prior of personal styles, forcing the personal representation ($c$) to align with local data distributions rather than the global average. During training, clients update their local decoders independently, while the shared encoders are aggregated by a central server using algorithms like FedAvg.
Visualizing Disentanglement
The effectiveness of FedDVA is most apparent when examining how it reconstructs data. By isolating universal and personal features, the model can "see" the underlying object independently of the user's specific style.
As shown in the figure above, when the model processes samples from different clients, it successfully maps them to "general features" (the digits themselves) and "personal features" (the specific noise or watermarks unique to that client). This disentanglement allows for a more explainable personalized model, where the model's internal "code" for a digit is separated from the "code" for the client's unique style.
Experimental Results
The researchers evaluated FedDVA on three primary benchmarks: Modified MNIST (with synthetic client marks), CelebA (attribute-biased face data), and CIFAR-10 (heterogeneous label distributions).
Key findings include:
- Superior Accuracy: FedDVA consistently outperformed vanilla FedAvg and state-of-the-art PFL methods like Ditto in classification tasks.
- Faster Convergence: The model reached peak accuracy in significantly fewer communication rounds, demonstrating the efficiency of learning disentangled representations.
- Robustness: Performance variance across clients was reduced, meaning the model provides a more consistent experience regardless of local data bias.
- Representation Quality: t-SNE visualizations revealed that universal representations ($z$) from all clients were well-mixed, while personalized representations ($c$) formed distinct clusters for each client, confirming the success of the disentanglement strategy.
Conclusion and Future Work
FedDVA provides a novel approach to PFL by focusing on representation disentanglement. By separating universal and personal traits, it achieves high performance and interpretability. However, challenges remain regarding the computational overhead of running dual encoders on edge devices and the sensitivity of the "slack" hyperparameter. Future research may explore applying this framework to more complex tasks like object detection or natural language processing.
Summary updated