FissionVAE: Federated Non-IID Image Generation with Latent Space and Decoder Decomposition

Chen Hu, Hanchi Ren, Jingjing Deng, Xianghua Xie, Xiaoke MaIJCAI 2025 · DOI

TL;DR

This paper introduces FissionVAE, a novel approach to federated image generation that tackles the challenge of non-IID (non-identically and independently distributed) data across clients. By decoupling the latent space and employing group-specific decoder branches, FissionVAE significantly improves image generation quality compared to existing federated VAE methods, producing sharper and less artifact-prone images.

What Problem Does This Paper Address?

Federated learning (FL) allows multiple clients to collaboratively train a machine learning model without sharing their private data. While FL has seen success in various domains, generating high-quality images in a federated setting, especially when client data is non-IID, remains a significant challenge. Non-IID data means that each client's dataset has a different distribution, which can lead to inconsistencies when models are aggregated. For generative models like Variational Autoencoders (VAEs), this can result in a shared latent space that is interpreted differently by each client's decoder, leading to blended features and artifacts in the generated images. Existing federated generative models, particularly Generative Adversarial Networks (GANs), have been explored, but VAEs, which offer a more direct probabilistic framework, have received less attention in this context. This paper aims to bridge this gap by proposing a VAE-based solution for federated non-IID image generation.

Methods

The core innovation of FissionVAE lies in its approach to handling non-IID data within a federated VAE framework. The authors propose two main strategies:

  • Latent Space Decoupling: Instead of forcing all clients to map their data to a single, unified latent distribution, FissionVAE decomposes the latent space. Each client group is assigned its own prior distribution for the latent variables. This allows encoders to map data from different groups to distinct regions of the latent space, preventing the averaging of dissimilar distributions during aggregation.
  • Group-Specific Decoder Branches: To further address the issue of blended features, FissionVAE introduces specialized decoder branches for each client group. While the encoder is aggregated globally, the decoders are aggregated only within their respective groups. This ensures that the decoding process is tailored to the specific data characteristics of each group, leading to more accurate reconstruction and generation.

The paper also explores hierarchical VAE architectures and various strategies for encoding the prior distributions of latent variables to enhance the decoupling process.

Datasets

The proposed FissionVAE was evaluated on two composite datasets:

  • Mixed MNIST: This dataset combines the MNIST (handwritten digits) and FashionMNIST (clothing items) datasets. It was split into two client groups, with 10 clients each, representing a binary non-IID scenario.
  • CHARM: This is a more diverse dataset comprising RGB images from five domains: cartoon faces, human faces, animals, remote sensing images, and marine vessels. This dataset was used to simulate a more realistic and challenging federated learning scenario with 5 client groups and a larger number of clients (20 per group).

Key Insights and Ideas

  • Latent space ambiguity is a major hurdle in federated VAEs: The paper highlights that simply aggregating VAEs trained on non-IID data leads to a "blending" of features, as decoders struggle to interpret a shared latent space consistently.
  • Decomposition is key: Both latent space decoupling and group-specific decoders independently improve generation quality, but their combination yields the best results. This suggests that addressing both the encoding and decoding stages is crucial for effective federated generation.
  • Heterogeneous decoders are feasible and beneficial: The FissionVAE architecture allows for clients to use decoder architectures of varying complexity, which is a practical advantage in federated settings where computational resources differ.
  • Prior distribution choice matters: The choice of prior distribution for the latent space can significantly impact performance, with "wave" encoding showing promise for distinguishing group priors.
  • Group-level privacy preservation: The paper suggests that by protecting the prior distributions of client groups, a form of group-level privacy can be achieved, as incorrect prior usage leads to mode collapse.

Key Results

  • FissionVAE significantly outperforms the baseline FedVAE on both Mixed MNIST and CHARM datasets in terms of generation quality, as measured by Fréchet Inception Distance (FID) and Inception Score (IS).
  • For instance, on the CHARM dataset, FissionVAE+L+D (combining latent space decoupling and decoder branches) achieved a lower FID (109.10) and higher IS (2.27) compared to FedVAE (167.18 FID, 1.57 IS) in federated settings.
  • The FissionVAE with decoder branches (+D) showed a notable reduction in visual blending artifacts compared to latent space decoupling alone (+L).
  • The hierarchical FissionVAE (FissionVAE+H+L+D) performed best on the more complex CHARM dataset, indicating its ability to capture intricate data distributions.

Limitations and Future Work

  • Performance on simpler datasets: The hierarchical FissionVAE showed a performance degradation on the simpler Mixed MNIST dataset, suggesting that the added complexity might introduce unnecessary redundancy without proportional gains.
  • Stability of heterogeneous decoders: While heterogeneous decoders are feasible, the paper notes that mismatches in architecture and weight space can disrupt VAE convergence, suggesting a need for further research into improving their stability.
  • Future directions: The authors suggest future work on improving the stability of heterogeneous decoder branches, enabling cross-modality data generation, and developing scalable strategies for handling a larger number of client groups in real-world federated learning scenarios.

Summary updated

BibTeX
@inproceedings{hu2025fissionvae,
  author = {Chen Hu and Hanchi Ren and Jingjing Deng and Xianghua Xie and Xiaoke Ma},
  title = {{FissionVAE: Federated Non-IID Image Generation with Latent Space and Decoder Decomposition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/597},
}
Figure