Abstract
Unsupervised image-to-image (I2I) translation has traditionally been limited to deterministic one-to-one mappings. XOGAN introduces a novel "XO-structure" that incorporates a latent variable $Z$ to capture style and texture, enabling diverse one-to-many generation from a single input without requiring paired training data. By enforcing cycle consistency across latent and image spaces, XOGAN effectively disentangles content from style, overcoming the common issue of mode collapse in generative models.
Introduction
Early breakthroughs in unsupervised image-to-image translation, such as CycleGAN and DiscoGAN, revolutionized the field by allowing models to learn mappings between domains without paired examples. However, these models are fundamentally deterministic: given a specific input, such as an edge map of a shoe, they consistently produce the same output.
In reality, image translation is often a "one-to-many" problem. A single sketch can correspond to a vast array of physical objects varying in color, material, and texture. Previous attempts to introduce stochasticity by adding noise often resulted in "mode collapse," where the generator learns to ignore the noise to minimize loss, reverting to a single "safe" average output.
Methodology: The XO-Structure
XOGAN addresses the deterministic barrier through a unique "XO" data flow architecture. The system utilizes three distinct generators:
- $G_B$: Maps an image from Domain $A$ (e.g., edges) and a latent variable $Z$ (representing style) to Domain $B$ (e.g., a photo).
- $G_A$: Performs the inverse mapping from Domain $B$ back to Domain $A$.
- $G_Z$: An encoder that extracts the latent style $Z$ from a real image in Domain $B$.
The model is trained using two primary paths to ensure robust learning:
- X-path (Forward): Starting with an image from $A$ and a random $Z$, the model generates a fake $B$, then uses $G_A$ and $G_Z$ to reconstruct the original $A$ and $Z$.
- O-path (Backward): Starting with a real image from $B$, the model decomposes it into its constituent $A$ (content) and $Z$ (style), then uses $G_B$ to reconstruct the original $B$.
By enforcing cycle consistency across these paths, the model is forced to encode all non-structural information (like color and texture) into the variable $Z$. This prevents the generator from ignoring the latent code, as $Z$ must be recoverable to satisfy the loss functions.
Experimental Setup and Datasets
The performance of XOGAN was evaluated on three major benchmarks, with training conducted in a strictly unsupervised (unpaired) manner:
- edges2shoes: Approximately 50,000 images of shoe edge maps and corresponding photos.
- edges2handbags: Approximately 140,000 images of handbag edge maps and photos.
- CelebA: A large-scale face dataset used for hair color translation tasks.
Key Insights and Results
XOGAN demonstrates several advantages over traditional unsupervised models:
- Diversity and Quality: Unlike baselines like UNIT or Noisy DiscoGAN, which often produce muted or repetitive results, XOGAN generates a wide spectrum of vibrant colors and textures for a single input.
- Disentanglement: The model successfully separates global style attributes from local structural content. When the latent variable $Z$ is fixed while the input sketch changes, the generated objects maintain a consistent "look" (e.g., same leather type and color).
- Style Transfer: Because $G_Z$ can extract style from any real image, users can perform "style swaps," applying the texture of one shoe to the shape of another.
Conclusion and Future Work
XOGAN represents a significant step forward in making unsupervised image translation more expressive and realistic. By moving beyond one-to-one mappings, it opens new possibilities for creative tools and data augmentation. Future research may explore more complex latent space priors, such as those used in Variational Autoencoders (VAEs), to capture even more intricate structural variations beyond color and texture.
Summary updated