Abstract
Generative Adversarial Networks (GANs), particularly the StyleGAN architecture, have set benchmarks in high-fidelity image synthesis. However, these models typically assume a unimodal data distribution, starting from a single learned "root constant." When applied to multi-modal datasets—such as a mixture of different animal species—this assumption leads to inefficiencies, mode collapse, and entangled representations. We introduce uMM-GAN (unsupervised Multi-Modal GAN), an architecture that learns $K$ distinct root constants to explicitly represent different data modes. Our results demonstrate that uMM-GAN significantly improves image quality (FID) and naturally disentangles high-level content (mode) from low-level style, even without supervision.
Introduction: The Challenge of Multi-Modal Synthesis
State-of-the-art GANs like StyleGAN generate images by transforming a fixed, learned tensor (the root constant) through a series of synthesis layers controlled by style vectors. While effective for homogeneous datasets (e.g., human faces), this "one-size-fits-all" starting point is ill-suited for datasets containing fundamentally different object classes or poses. Forcing a single template to warp into a cat, a dog, or a car requires the mapping network to learn highly complex, non-linear deformations. This often results in "hybrid" artifacts, where features of different modes are blended inappropriately, and a lack of control over the semantic category of the generated image.
Methodology: uMM-GAN Architecture
The core innovation of uMM-GAN is the transition from a single root constant to a set of $K$ learnable templates. This allows the generator to choose a starting point that is already close to the target mode's manifold.
As shown in the architecture diagram above, the process involves:
- Latent Mapping: A random vector $z$ is processed by a mapping network to produce two outputs: a style vector $w$ and a set of mixture weights $w_R$ (via a softmax layer).
- Mode Mixing: The weights $w_R$ are used to linearly combine $K$ independent root constants into a single mixed constant. This mixed constant serves as the "content" or "mode" seed.
- Style Injection: The mixed constant is fed into the synthesis network $G$. At each layer, the style vector $w$ is injected via Adaptive Instance Normalization (AdaIN) to control visual attributes like color, texture, and lighting.
By separating the selection of the root constant from the stylistic rendering, uMM-GAN achieves unsupervised disentanglement. One can modify the mode by changing $w_R$ while keeping $w$ constant, effectively "re-styling" the same content or changing the content while preserving the style.
Experimental Evaluation
The authors evaluated uMM-GAN across several domains, including synthetic 2D distributions, handwritten characters, and natural image datasets like DRIT Cats & Dogs and LSUN Bedrooms.
Key Results
- Quantitative Performance: uMM-GAN consistently achieves lower Fréchet Inception Distance (FID) scores than standard StyleGAN. On the Cats & Dogs dataset, uMM-GAN reached an FID of 117.25, outperforming the baseline StyleGAN (126.90).
- Synergy in Multi-Modal Training: Interestingly, a single uMM-GAN model trained on a combined dataset of cats and dogs outperformed two separate StyleGAN models trained on each class individually. This suggests that the shared synthesis layers in uMM-GAN effectively learn common stylistic features across modes, while the multiple root constants handle the class-specific structural differences.
- Scalability: Increasing the number of modes $K$ generally leads to better data approximation. In the LSUN Bedrooms dataset, increasing $K$ from 1 to 16 reduced the FID from 189.53 to 157.01.
Key Insights
- Disentanglement by Design: The architecture enforces a hierarchy where the root constant defines the "what" (mode) and the AdaIN parameters define the "how" (style).
- Unsupervised Clustering: Without any labels, the model naturally clusters data into meaningful modes. For example, in a car dataset, the learned modes often correspond to different camera angles or vehicle types.
- Computational Efficiency: Providing multiple starting points is more efficient than forcing a mapping network to perform extreme geometric transformations on a single point.
Conclusion and Future Work
uMM-GAN provides a simple yet powerful modification to the StyleGAN framework, enabling it to handle complex, multi-modal datasets without supervision. While the current implementation requires a pre-defined hyperparameter $K$, future research could explore methods to automatically determine the optimal number of modes during training. Additionally, extending this multi-modal approach to multiple hierarchical levels could allow for even finer control over generated content.
Summary updated