Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators

Daniel Stoller, Sebastian Ewert, Simon Dixon

TL;DR

FactorGAN introduces a principled way to train Generative Adversarial Networks (GANs) using a mix of complete (paired) and incomplete (unpaired) data. By factorizing the discriminator into "marginal" and "dependency" sub-units, the model can leverage large amounts of easily available unpaired data to improve performance on tasks where paired labels are scarce, such as image segmentation and audio source separation.

What Problem Does This Paper Address?

Standard GANs are notoriously data-hungry. To learn a joint distribution—for example, the relationship between a street scene and its corresponding segmentation map—they typically require a large dataset of perfectly paired examples.

In the real world, paired data is expensive and rare. We might have millions of street photos and thousands of segmentation maps, but only a few hundred where the two are matched. While frameworks like CycleGAN can learn from unpaired data, they often struggle to incorporate the small amount of paired data we do have in a mathematically consistent way. They rely on complex "cycle-consistency" losses that don't always guarantee the generator will converge to the true target distribution.

The authors ask: Can we build a GAN that uses every scrap of data available, whether it's paired or not, using a single, unified adversarial objective?

Methods

The core innovation is FactorGAN, which replaces the single "monolithic" discriminator with a set of specialized factorised discriminators.

The researchers mathematically decompose the "joint density ratio" (the signal the discriminator uses to guide the generator) into two parts:

  1. Marginal Discriminators: These look at individual components of the data (e.g., just the image or just the segmentation map). Because they only look at one part, they can be trained on any available data, even if it's unpaired.
  2. Dependency Discriminators ($c_P$ and $c_Q$): These focus specifically on the relationship between the parts.
  3. The $c_P$ discriminator learns what real-world dependencies look like (requiring paired data).
  4. The $c_Q$ discriminator identifies dependencies created by the generator (trained by shuffling generated samples to break their internal correlations).

By combining the outputs of these sub-units using a specific formula derived from first principles, FactorGAN reconstructs an estimate of the full joint distribution. This allows the generator to receive high-quality feedback even when paired training samples are extremely limited.

Datasets

The authors evaluated FactorGAN across three distinct domains:

  • Paired MNIST: A synthetic dataset of vertically stacked digit pairs used to control and test dependency modeling.
  • Image-to-Image Translation:
    • Cityscapes: Real-world street scenes and their semantic segmentation maps.
  • Edges2Shoes: Sketches of shoes paired with photos of the actual shoes.
  • Audio Source Separation:
    • MUSDB: A standard dataset for separating music into vocals and accompaniment.

Key Insights and Ideas

  • Divide and Conquer: You don't need a single network to understand everything at once. By splitting the discriminator's job, you can train the "easy" parts (marginals) on massive unpaired datasets and save the "hard" parts (dependencies) for the limited paired data.
  • Principled Fusion: Unlike previous "hybrid" models that just add different loss functions together and hope for the best, FactorGAN provides a theoretical proof that their combined discriminator approximates the optimal discriminator of a standard GAN.
  • Architectural Flexibility: Because the discriminator is modular, you can bake "prior knowledge" into specific sub-units. For example, if you know certain features don't affect dependencies, you can simplify the dependency discriminator to prevent overfitting.

Key Results

FactorGAN consistently outperformed standard GANs and CycleGAN in low-data scenarios, demonstrating significant improvements in both quantitative metrics and visual quality.

FID scores across different numbers of paired samples. FactorGAN achieves significantly lower (better) FID than standard GANs when paired data is scarce.
  • Massive Accuracy Gains: On the Cityscapes segmentation task, FactorGAN achieved a 14.9% absolute improvement in accuracy over CycleGAN using only 25 paired examples.
  • Sample Efficiency and Diversity: In the Edges2Shoes task, FactorGAN produced high-fidelity, diverse images with just 100 paired samples. In contrast, standard GANs suffered from severe "mode collapse," repeatedly producing the exact same outputs regardless of the input sketch.
Images generated by FactorGAN on the Edges2Shoes dataset, showing high diversity and fidelity even with limited paired training data.
  • Cross-Domain Success: The method successfully transferred to audio, providing better source separation (SDR) than standard GANs by better modeling the relationship between vocal and instrumental tracks.

Limitations and Future Work

  • Dependency Overfitting: While FactorGAN is efficient, the dependency discriminator ($c_P$) still requires some paired data. If paired data is virtually non-existent, $c_P$ can still overfit, leading to unstable training.
  • Training Mismatch: The authors noted a potential "mismatch" where the dependency discriminator is trained on real data but evaluated on fake data. They suggest that better uncertainty calibration (e.g., using Dropout) might help mitigate this.
  • Complex Partitioning: The paper focuses on simple splits (e.g., two parts). Future work could explore more complex, hierarchical factorizations for high-dimensional data like video or multi-channel sensor arrays.

Summary updated

Abstract

Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting. However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels. Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance. To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies. This allows splitting the discriminator in a GAN into multiple "sub-discriminators" that can be independently trained from incomplete observations. Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework. We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available. For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 14.9% over CycleGAN while using only 25 additional paired examples.
BibTeX
@inproceedings{stoller2020training,
  author = {Daniel Stoller and Sebastian Ewert and Simon Dixon},
  title = {{Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure