Kernel of CycleGAN as a Principle homogeneous space

Nikita Moriakov, Jonas Adler, Jonas TeuwenarXiv 2020 · arXiv

TL;DR

This paper mathematically proves that the CycleGAN loss function is inherently ambiguous: it admits a vast space of "perfect" solutions that are semantically incorrect. By treating the solution space as a principal homogeneous space, the authors show that any valid translation can be transformed into an equally "valid" but useless one via hidden symmetries (automorphisms) in the data, a phenomenon they demonstrate empirically using MNIST and medical imaging datasets.

What Problem Does This Paper Address?

CycleGAN is a powerhouse for unpaired image-to-image translation (e.g., turning horses into zebras or MRIs into CT scans). It works by enforcing two main things: distribution matching (the output should look like the target domain) and cycle consistency (translating back and forth should return the original image).

However, researchers have long suspected that these constraints aren't enough to guarantee a unique, semantically correct mapping. For example, if you're translating between two identical domains, the "correct" answer is the identity mapping. But could the network learn to swap labels instead? This paper formalizes this "identifiability" problem, proving that the CycleGAN loss function has a massive "kernel" (a null space of solutions) that allows for unwanted transformations that still satisfy all mathematical constraints.

Methods

The authors employ tools from group theory and measure theory to analyze the CycleGAN objective.

  1. Theoretical Analysis of the Kernel: They define the "kernel" as the set of all mapping pairs $(G, F)$ that achieve zero loss. They prove that if a domain has a non-trivial symmetry (a probability space automorphism), you can compose any valid solution with that symmetry to get a new, mathematically perfect solution.
  2. Principal Homogeneous Space: They show that the group of these symmetries acts "freely and transitively" on the space of exact solutions. In layman's terms: for any two perfect solutions, there is exactly one symmetry that connects them.
  3. Perturbation Bounds: Since real-world models rarely reach exactly zero loss, the authors provide bounds for "approximate" solutions, including the "extended" CycleGAN loss which adds an identity mapping term to discourage drastic changes.
  4. Empirical Validation: They test these theories by training a CycleGAN to perform an identity task (MNIST-to-MNIST) to see if it learns the identity mapping or falls into a "permuted" trap.
Figure 1: CycleGAN model.

Datasets

  • MNIST: The classic handwritten digit dataset. Used here for a "toy" experiment where the source and target domains are identical.
  • BRATS2015: A medical imaging dataset containing brain MRIs (T1 and Flair modalities). This represents a high-stakes real-world application where "hallucinated" or shifted features could be dangerous.

Key Insights and Ideas

  • Symmetry is the Culprit: If your data distribution is symmetric (e.g., flipping a brain scan horizontally results in a valid-looking brain scan), CycleGAN cannot distinguish between the "true" mapping and the "flipped" mapping.
  • The Identity Loss Isn't a Total Cure: While adding an identity loss term ($\alpha_{id}$) helps reduce "tint shift," the authors show it doesn't eliminate the ambiguity, especially when translating between very different modalities where the identity term must be kept small to allow any translation at all.
  • Neural Networks Learn Permutations: In the MNIST-to-MNIST task, the network didn't learn the identity mapping. Instead, it learned to consistently swap digits (e.g., mapping a '2' to a '6'). Because the distribution of digits remained the same, the loss stayed low, even though the result was semantically wrong.

Key Results

  • MNIST Permutations: The confusion matrices in Figure 3/5 clearly show that the generators learned deterministic permutations of digit classes rather than the identity mapping. For instance, the A2B generator might map class 2 to class 6 with high probability.
Figure 3: Normalized confusion matrices for A2B and B2A generator respectively.
  • Medical Imaging Symmetries: On the BRATS2015 dataset, the authors found that horizontal flipping is a distribution symmetry that the model exploits. While increasing the weight of the identity loss ($\alpha_{id}$) reduced this symmetry, it didn't necessarily lead to better image quality (measured by PSNR), highlighting the difficulty of balancing these constraints.

Limitations and Future Work

The paper primarily focuses on the existence of these problematic solutions rather than providing a "silver bullet" fix. The authors suggest that the specific architecture of the neural network (inductive biases) might be the only thing currently preventing CycleGAN from failing more often in practice.

Future work could involve:

  • Developing new loss terms that explicitly penalize these hidden symmetries.
  • Exploring "multimodal" translation models that acknowledge and model the inherent uncertainty in unpaired tasks.
  • Investigating how architectural choices (like skip connections or bottleneck sizes) influence which solution in the kernel the network eventually settles on.

Summary updated

Abstract

Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions. We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions. We examine the case of zero `pure' CycleGAN loss first in its generality, and, subsequently, expand our analysis to approximate solutions for `extended' CycleGAN loss where identity loss term is included. In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist. Furthermore, we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results. We conclude that finding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result utterly useless.
BibTeX
@misc{moriakov2020kernel,
  author = {Nikita Moriakov and Jonas Adler and Jonas Teuwen},
  title = {{Kernel of CycleGAN as a Principle homogeneous space}},
  year = {2020},
  eprint = {2001.09061},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09061},
}
Figure