Unsupervised K-modal styled content generation

Omry Sendik, Dani Lischinski, Daniel Cohen-Or

Abstract

Generative Adversarial Networks (GANs), particularly the StyleGAN architecture, have set benchmarks in high-fidelity image synthesis. However, these models typically assume a unimodal data distribution, starting from a single learned "root constant." When applied to multi-modal datasets—such as a mixture of different animal species—this assumption leads to inefficiencies, mode collapse, and entangled representations. We introduce uMM-GAN (unsupervised Multi-Modal GAN), an architecture that learns $K$ distinct root constants to explicitly represent different data modes. Our results demonstrate that uMM-GAN significantly improves image quality (FID) and naturally disentangles high-level content (mode) from low-level style, even without supervision.

Introduction: The Challenge of Multi-Modal Synthesis

State-of-the-art GANs like StyleGAN generate images by transforming a fixed, learned tensor (the root constant) through a series of synthesis layers controlled by style vectors. While effective for homogeneous datasets (e.g., human faces), this "one-size-fits-all" starting point is ill-suited for datasets containing fundamentally different object classes or poses. Forcing a single template to warp into a cat, a dog, or a car requires the mapping network to learn highly complex, non-linear deformations. This often results in "hybrid" artifacts, where features of different modes are blended inappropriately, and a lack of control over the semantic category of the generated image.

Methodology: uMM-GAN Architecture

The core innovation of uMM-GAN is the transition from a single root constant to a set of $K$ learnable templates. This allows the generator to choose a starting point that is already close to the target mode's manifold.

The uMM-GAN architecture: Latent vector $z$ is mapped to style $w$ and mode mixture weights $w_R$, which combine $K$ root constants into a starting state for generator $G$.

As shown in the architecture diagram above, the process involves:

  1. Latent Mapping: A random vector $z$ is processed by a mapping network to produce two outputs: a style vector $w$ and a set of mixture weights $w_R$ (via a softmax layer).
  2. Mode Mixing: The weights $w_R$ are used to linearly combine $K$ independent root constants into a single mixed constant. This mixed constant serves as the "content" or "mode" seed.
  3. Style Injection: The mixed constant is fed into the synthesis network $G$. At each layer, the style vector $w$ is injected via Adaptive Instance Normalization (AdaIN) to control visual attributes like color, texture, and lighting.

By separating the selection of the root constant from the stylistic rendering, uMM-GAN achieves unsupervised disentanglement. One can modify the mode by changing $w_R$ while keeping $w$ constant, effectively "re-styling" the same content or changing the content while preserving the style.

Experimental Evaluation

The authors evaluated uMM-GAN across several domains, including synthetic 2D distributions, handwritten characters, and natural image datasets like DRIT Cats & Dogs and LSUN Bedrooms.

Key Results

  • Quantitative Performance: uMM-GAN consistently achieves lower Fréchet Inception Distance (FID) scores than standard StyleGAN. On the Cats & Dogs dataset, uMM-GAN reached an FID of 117.25, outperforming the baseline StyleGAN (126.90).
  • Synergy in Multi-Modal Training: Interestingly, a single uMM-GAN model trained on a combined dataset of cats and dogs outperformed two separate StyleGAN models trained on each class individually. This suggests that the shared synthesis layers in uMM-GAN effectively learn common stylistic features across modes, while the multiple root constants handle the class-specific structural differences.
  • Scalability: Increasing the number of modes $K$ generally leads to better data approximation. In the LSUN Bedrooms dataset, increasing $K$ from 1 to 16 reduced the FID from 189.53 to 157.01.
High-fidelity images of cats and dogs generated by uMM-GAN, demonstrating its ability to capture multiple distinct modes in an unsupervised manner.

Key Insights

  • Disentanglement by Design: The architecture enforces a hierarchy where the root constant defines the "what" (mode) and the AdaIN parameters define the "how" (style).
  • Unsupervised Clustering: Without any labels, the model naturally clusters data into meaningful modes. For example, in a car dataset, the learned modes often correspond to different camera angles or vehicle types.
  • Computational Efficiency: Providing multiple starting points is more efficient than forcing a mapping network to perform extreme geometric transformations on a single point.

Conclusion and Future Work

uMM-GAN provides a simple yet powerful modification to the StyleGAN framework, enabling it to handle complex, multi-modal datasets without supervision. While the current implementation requires a pre-defined hyperparameter $K$, future research could explore methods to automatically determine the optimal number of modes during training. Additionally, extending this multi-modal approach to multiple hierarchical levels could allow for even finer control over generated content.

Summary updated

Abstract

The emergence of deep generative models has recently enabled the automatic generation of massive amounts of graphical content, both in 2D and in 3D. Generative Adversarial Networks (GANs) and style control mechanisms, such as Adaptive Instance Normalization (AdaIN), have proved particularly effective in this context, culminating in the state-of-the-art StyleGAN architecture. While such models are able to learn diverse distributions, provided a sufficiently large training set, they are not well-suited for scenarios where the distribution of the training data exhibits a multi-modal behavior. In such cases, reshaping a uniform or normal distribution over the latent space into a complex multi-modal distribution in the data domain is challenging, and the generator might fail to sample the target distribution well. Furthermore, existing unsupervised generative models are not able to control the mode of the generated samples independently of the other visual attributes, despite the fact that they are typically disentangled in the training data. In this paper, we introduce uMM-GAN, a novel architecture designed to better model multi-modal distributions, in an unsupervised fashion. Building upon the StyleGAN architecture, our network learns multiple modes, in a completely unsupervised manner, and combines them using a set of learned weights. We demonstrate that this approach is capable of effectively approximating a complex distribution as a superposition of multiple simple ones. We further show that uMM-GAN effectively disentangles between modes and style, thereby providing an independent degree of control over the generated content.
BibTeX
@article{sendik2020unsupervised,
  author = {Omry Sendik and Dani Lischinski and Daniel Cohen-Or},
  title = {{Unsupervised K-modal styled content generation}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3386569.3392454},
}
Figure