Abstract
Recent generative adversarial networks (GANs) are able to generate impressive photo-realistic images. However, controllable generation with GANs remains an open research problem. Achieving controllable generation requires semantically interpretable and disentangled factors of variation. It is challenging to achieve this goal using simple fixed distributions such as Gaussian distribution. Instead, we propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Self-training provides an iterative feedback in the GAN training, from the discriminator to the generator, and progressively improves the proposal of the latent codes as training proceeds. The latent codes are sampled from a latent variable model that is learned in the feature space of the discriminator. We consider a normalized independent component analysis model and learn its parameters through tensor factorization of the higher-order moments. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder, and is able to discover semantically meaningful latent codes without any supervision. We empirically demonstrate on both cars and faces datasets that each group of elements in the learned code controls a mode of variation with a semantic meaning, e.g. pose or background change. We also demonstrate with quantitative metrics that our method generates better results compared to other approaches.
Introduction
Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity images. However, a persistent challenge remains: controllable generation. In standard GANs, the generator is typically fed a random vector from a simple, fixed distribution like a Gaussian. This lack of structure means that individual dimensions of the latent space do not correspond to interpretable semantic attributes, such as "pose" or "color." Achieving such control usually requires large-scale labeled datasets, which are expensive or unavailable for many domains.
We introduce ST-GAN (Self-Training GAN), a framework that discovers semantically meaningful factors of variation in an entirely unsupervised manner. By replacing fixed latent distributions with a learnable model and employing a self-training loop, ST-GAN enables precise control over image attributes without any human supervision.
Methodology
The ST-GAN architecture moves away from the standard practice of using fixed noise distributions. Instead, it treats the latent space as a learnable component that evolves alongside the generator and discriminator.
Latent Variable Model (LVM) via ICA
The core of the framework is a "Normalized Independent Component Analysis" (ICA) model. This model assumes that the complex features of an image are linear combinations of independent, underlying signals. By sampling latent codes from this structured model rather than a Gaussian, the generator is forced to learn a more disentangled representation.
Tensor Factorization and Self-Training
To learn the parameters of the ICA model without labels, ST-GAN leverages the discriminator's internal representations. The discriminator naturally learns to identify semantic structures to distinguish real from fake images. We calculate higher-order moments (cumulants) of these features and apply tensor factorization to decompose them. This process "unmixes" the features into independent directions representing semantic attributes.
This creates a self-training loop: as the GAN trains, the discriminator's features become more refined. These improved features are used to update the LVM, which then provides more structured latent codes back to the generator, progressively improving the system's controllability.
Masking Loss for Disentanglement
To further ensure that each "knob" controls only one attribute (disentanglement), we introduce a masking loss. By perturbing a single element of the latent code and requiring the model to predict which element was changed based on the output image, we encourage the model to assign distinct, consistent effects to each latent dimension.
Experiments and Results
The framework was evaluated on the CelebA (faces) and Cars datasets at a 64x64 resolution.
Qualitative Results
As shown in the figure below, ST-GAN successfully discovers independent factors of variation. By interpolating along a single dimension of the learned latent code, we can smoothly transition attributes like head pose, hair color, or car orientation without affecting other aspects of the image.
Quantitative Performance
ST-GAN significantly outperforms unsupervised baselines in controllability metrics while maintaining high image quality.
- Controllability: On CelebA, it achieved a Mean Absolute Error (MAE) of 0.265 and an LPIPS score of 0.040, indicating that latent changes lead to more distinct and predictable perceptual shifts than in standard SNGAN or VAE models.
- Image Quality: The model maintains competitive realism, with Frechet Inception Distance (FID) scores of 25.21 on CelebA and 12.96 on Cars.
Discussion and Conclusion
The success of ST-GAN highlights two key insights: first, the discriminator is a rich source of semantic information that can be extracted via tensor factorization; second, moving beyond simple Gaussian priors to data-driven, learnable latent distributions is essential for unsupervised control.
While the current work focuses on 64x64 resolutions and requires careful initialization (a "warm-up" period), it provides a robust foundation for unsupervised disentanglement. Future work will explore scaling these techniques to high-resolution imagery and applying more complex latent variable models to capture abstract or rare semantic attributes.
Summary updated