Abstract
Recently the Generative Adversarial Network has become a hot topic. Considering the application of GAN in multi-user environment, we propose Distributed-GAN. It enables multiple users to train with their own data locally and generates more diverse samples. Users don't need to share data with each other to avoid the leakage of privacy. In recent years, commercial companies have launched cloud platforms based on artificial intelligence to provide model for users who lack computing power. We hope our work can inspire these companies to provide more powerful AI services.
Introduction
Generative Adversarial Networks (GANs) have revolutionized data augmentation and transfer learning, but they typically require massive, centralized datasets to perform well. In the era of cloud-based AI services, users often have to upload their private data to a central server to train these models, creating a significant privacy risk.
Distributed-GAN is a framework that allows multiple users to collaboratively train GANs while keeping their sensitive raw data local. By leveraging distributed and federated learning techniques, this approach protects privacy and can significantly reduce training time compared to traditional, centralized GANs.
Methods and Architecture
The core innovation of Distributed-GAN is the realization that a generator doesn't need to see real images to learn; it only needs to hear how "fake" its own images look from a discriminator that has seen real images. By sharing these critiques—either as gradients or scores—instead of raw images, privacy is maintained.
The paper introduces three specific approaches to this distributed setup:
- The Federated Approach: Users train local discriminators and upload a portion of their gradients to a central server. The server aggregates these to update a global discriminator, which then trains a central generator.
- The Weighted Output Approach: The generator receives feedback based on the "consensus" (average output) of multiple independent discriminators.
- The Naive Distributed Approach: A single generator is trained against multiple discriminators sequentially or simultaneously within a single epoch, learning to "fool" each user's local discriminator one by one.
Experimental Evaluation
The authors evaluated Distributed-GAN using three benchmarks: MNIST (handwritten digits), CelebA (celebrity faces), and LSUN (scene understanding). To simulate a distributed environment, datasets were split so that different "users" held different subsets (e.g., User A has digits 0-4, User B has 5-9).
Key Results
The models successfully generated realistic handwritten digits and celebrity faces across all three distributed methods. Even when users held entirely different classes of data, the central generator was able to learn the entire distribution.
Beyond qualitative success, the distributed nature of these algorithms offers a performance boost. By parallelizing the "critiquing" process across multiple discriminators, the total training time was consistently lower than traditional centralized training for both MNIST and CelebA.
Discussion and Insights
A critical finding involves the "Weighted Output" approach: its effectiveness depends heavily on domain similarity. It works well when users have similar data (like digits 6 and 8), but can fail if domains are too disparate (like 4 and 7), as averaged feedback may confuse the generator.
Limitations and Future Work
Despite its promise, several hurdles remain for Distributed-GAN:
- Theoretical Grounding: Evidence is currently primarily empirical; a rigorous mathematical derivation for the convergence of these distributed adversarial loops is needed.
- Model Collapse: Like standard GANs, Distributed-GAN remains susceptible to "model collapse," where the generator produces a limited set of outputs.
- Communication and Security: Real-world implementation requires optimizing communication overhead. Furthermore, while raw data is not shared, the system must be hardened against "inference attacks" that might extract information from shared gradients or model parameters.
Summary updated