Partial Disentanglement with Partially-Federated GANs (PaDPaF)

Abdulla Jasem Almansoori, Samuel Horváth, Martin Takác

Abstract

The Partial Disentanglement with Partially-Federated GANs (PaDPaF) framework addresses the challenge of training generative models across decentralized clients with heterogeneous data. By partitioning the model into shared global "content" modules and private local "style" modules, PaDPaF enables high-quality personalized data generation and robust feature extraction. The approach utilizes latent-contrastive regularization to ensure effective disentanglement, allowing for applications such as style transfer and data anonymization even in highly non-IID settings.

Introduction

Federated Learning (FL) typically focuses on classification tasks, but extending it to generative models like GANs is complicated by data heterogeneity. Variations in lighting, rotation, or sensor noise—collectively referred to as "style"—can cause local data distributions to diverge significantly. Standard FL methods often struggle with this non-IID data, leading to poor generalization or the loss of client-specific nuances.

PaDPaF tackles this by aiming for disentanglement: separating the core semantic content (e.g., the identity of a digit) from its specific stylistic manifestation. The goal is to learn a shared content representation across the federation while respecting the unique stylistic characteristics of individual clients without centralizing their private data.

Methods

The PaDPaF architecture splits the GAN components into Federated Modules, which are averaged globally (e.g., via FedAvg), and Private Modules, which remain local to each client.

Architecture and Federation

The "content" portions of the generator and discriminator are shared, while "style" components—such as style vectorizers and conditional batch normalization layers—are kept private. This allows the global model to learn universal features while local models specialize in client-specific rendering.

Architecture of the PaDPaF framework showing the split between federated content and private style modules.

Latent-Contrastive Regularization

To enforce the separation of content and style, the authors employ a regularization term inspired by Barlow Twins. This loss encourages the discriminator to produce consistent representations for images sharing the same content latent, regardless of the style latent used during generation.

Causal Foundation

The framework is grounded in a causal model where client identity influences both content and style. Through federated averaging, the client-specific influence on the content mechanism is effectively marginalized, leaving a robust, global content representation.

Causal graph representing the dependencies between client identity, content, style, and observed data.

Experimental Evaluation

The model was tested across several datasets designed to simulate real-world heterogeneity, including MNIST, CIFAR-10, and CelebA. In these setups, clients were assigned unique data augmentations or specific attributes (like "Mustaches" or "Eyeglasses") to represent distinct local styles.

Key Results

  • Feature Quality: A linear classifier trained on unsupervised content features from PaDPaF achieved 98.1% accuracy on MNIST, indicating high-quality disentanglement.
  • Generative Performance: PaDPaF achieved an average Frechet Inception Distance (FID) of 8.28, significantly outperforming baselines like Ditto (60.46).
  • Disentanglement Verification: Style features were highly predictive of client identity (93.3% accuracy), whereas content features were not, confirming successful separation.

Discussion and Future Work

PaDPaF demonstrates that partial federation is a powerful tool for handling non-IID data in generative modeling. It enables "style transfer" and data anonymization by re-rendering content in a different client's style.

However, limitations remain. The current architecture sometimes struggles with geometric transformations like rotations, occasionally conflating them with content. Future research may explore more flexible generator architectures or server-side deployment strategies to further enhance privacy and robustness.

Summary updated

BibTeX
@article{almansoori2023partial,
  author = {Abdulla Jasem Almansoori and Samuel Horváth and Martin Takác},
  title = {{Partial Disentanglement with Partially-Federated GANs (PaDPaF)}},
  journal = {Accepted by TMLR},
  year = {2023},
  url = {https://openreview.net/forum?id=vsez76EAV8},
}
Figure