Federated CycleGAN for Privacy-Preserving Image-to-Image Translation

Joonyoung Song, Jong Chul YearXiv 2021 · arXiv · OpenReview

Abstract

Unsupervised image-to-image translation methods such as CycleGAN learn to convert images from one domain to another using unpaired training data sets from different domains. Unfortunately, these approaches still require centrally collected unpaired records, potentially violating privacy and security issues. Although the recent federated learning (FL) allows a neural network to be trained without data exchange, the basic assumption of the FL is that all clients have their own training data from a similar domain, which is different from our image-to-image translation scenario in which each client has images from its unique domain and the goal is to learn image translation between different domains without accessing the target domain data. To address this, here we propose a novel federated CycleGAN architecture that can learn image translation in an unsupervised manner while maintaining the data privacy. Specifically, our approach arises from a novel observation that CycleGAN loss can be decomposed into the sum of client specific local objectives that can be evaluated using only their data. This local objective decomposition allows multiple clients to participate in federated CycleGAN training without sacrificing performance. Furthermore, our method employs novel switchable generator and discriminator architecture using Adaptive Instance Normalization (AdaIN) that significantly reduces the band-width requirement of the federated learning. Our experimental results on various unsupervised image translation tasks show that our federated CycleGAN provides comparable performance compared to the non-federated counterpart.

Introduction

Unsupervised image-to-image (I2I) translation, popularized by models like CycleGAN, typically requires a central server to hold large collections of unpaired images from two different domains (e.g., "Photos" and "Paintings"). However, in the real world, data is often siloed due to privacy, security, or copyright concerns. For instance, a hospital might possess sensitive medical scans, while a research lab holds corresponding denoised targets, yet neither can share their raw data.

Comparison between centralized training (left) where private data is shared, and federated training (right) where only local gradients are exchanged.

Standard Federated Learning (FL) addresses privacy by training models locally and sharing only gradients. However, standard FL assumes homogeneous data across clients. In I2I translation, the problem is inherently heterogeneous: Client A possesses Domain X, and Client B possesses Domain Y. Training a CycleGAN in this "split-domain" federated setting was previously an open challenge because the loss functions for one domain typically require samples from the other to calculate.

Methodology

To realize Federated CycleGAN (FedCycleGAN), two primary innovations were introduced:

1. Local Objective Decomposition

The researchers observed that the total CycleGAN loss—comprising adversarial loss and cycle-consistency loss—can be mathematically decomposed into two domain-specific local objectives.

  • Client X calculates gradients using only Domain X data.
  • Client Y calculates gradients using only Domain Y data.
  • The Server aggregates these gradients to update the global model.

This allows the model to learn the relationship between domains without any single entity accessing both datasets.

2. Switchable Architecture (AdaIN)

Standard CycleGAN requires two generators and two discriminators, which creates a significant communication bottleneck in a federated setting. To mitigate this, the authors developed a Switchable CycleGAN architecture. Instead of dual networks, they utilize a single shared generator and discriminator. By employing Adaptive Instance Normalization (AdaIN) codes, the network "switches" its behavior to handle different domains. This reduces the number of transmitted parameters by nearly 50% (from ~69M to ~35M).

Architecture comparison: (a) Standard CycleGAN with dual networks vs. (b) Switchable FedCycleGAN using a shared generator/discriminator with AdaIN modulation.

Experimental Results

The framework was evaluated on two distinct tasks:

  • Style Transfer: Tasks included Summer-to-Winter, Photo-to-Van Gogh, and Horse-to-Zebra using standard unpaired datasets.
  • Medical Imaging: Low-dose Computed Tomography (CT) denoising using the AAPM 2016 Low Dose CT Grand Challenge dataset, highlighting the model's utility for sensitive medical data.

Key Findings

  • Performance Parity: FedCycleGAN performs almost identically to centralized training. In the Summer-to-Winter task, FedCycleGAN achieved a Frechet Inception Distance (FID) of 74.72, slightly outperforming the centralized baseline (75.40). For CT scans, the federated model achieved a Peak Signal-to-Noise Ratio (PSNR) of 37.14 dB, matching the centralized baseline of 37.11 dB.
  • Communication Efficiency: The switchable architecture successfully reduced trainable parameters from 69.5 million to 35.5 million with negligible impact on image quality.
  • Scalability: The decomposition of the global gradient into a sum of local gradients allows the system to scale to multiple clients per domain by averaging their respective domain-specific updates.

Discussion and Future Work

While FedCycleGAN preserves data privacy by avoiding raw data exchange, the authors note that "gradient inversion" attacks remain a theoretical risk. Future iterations may incorporate Differential Privacy (DP) to provide stronger mathematical guarantees. Additionally, moving from synchronous to asynchronous updates would improve robustness for real-world edge computing scenarios where client availability fluctuates. Finally, exploring domain imbalance—where one client has significantly more data than another—remains an important area for future research.

Summary updated

Abstract

Unsupervised image-to-image translation methods such as CycleGAN learn to convert images from one domain to another using unpaired training data sets from different domains. Unfortunately, these approaches still require centrally collected unpaired records, potentially violating privacy and security issues. Although the recent federated learning (FL) allows a neural network to be trained without data exchange, the basic assumption of the FL is that all clients have their own training data from a similar domain, which is different from our image-to-image translation scenario in which each client has images from its unique domain and the goal is to learn image translation between different domains without accessing the target domain data. To address this, here we propose a novel federated CycleGAN architecture that can learn image translation in an unsupervised manner while maintaining the data privacy. Specifically, our approach arises from a novel observation that CycleGAN loss can be decomposed into the sum of client specific local objectives that can be evaluated using only their data. This local objective decomposition allows multiple clients to participate in federated CycleGAN training without sacrificing performance. Furthermore, our method employs novel switchable generator and discriminator architecture using Adaptive Instance Normalization (AdaIN) that significantly reduces the band-width requirement of the federated learning. Our experimental results on various unsupervised image translation tasks show that our federated CycleGAN provides comparable performance compared to the non-federated counterpart.
BibTeX
@misc{song2021federated,
  author = {Joonyoung Song and Jong Chul Ye},
  title = {{Federated CycleGAN for Privacy-Preserving Image-to-Image Translation}},
  year = {2021},
  eprint = {2106.09246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.09246},
}
Figure