Abstract
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing class-specific information using conditional Batch Normalization (cBN). In this paper we propose to generalize both BN and cBN using a Whitening and Coloring based batch normalization. We show that our conditional Coloring can represent categorical conditioning information which largely helps the cGAN qualitative results. Moreover, we show that full-feature whitening is important in a general GAN scenario in which the training process is known to be highly unstable. We test our approach on different datasets and using different GAN networks and training protocols, showing a consistent improvement in all the tested frameworks. Our CIFAR-10 conditioned results are higher than all previous works on this dataset.
Introduction
Training Generative Adversarial Networks (GANs) is notoriously unstable due to the adversarial nature of finding a Nash equilibrium between a generator and a discriminator. While Batch Normalization (BN) has become a standard tool for stabilizing deep networks by standardizing features dimension-by-dimension, it fundamentally ignores the correlations between different feature dimensions. This paper introduces the Whitening and Coloring (WC) batch transform, a generalization of BN and conditional BN (cBN). By fully decorrelating features (whitening) and then re-projecting them using learned multivariate filters (coloring), the WC transform significantly stabilizes training and enhances the expressiveness of conditional GANs.
Methods
The proposed Whitening and Coloring (WC) transform replaces standard BN in the generator network with a two-step process:
- Whitening via Cholesky Decomposition: Unlike traditional ZCA whitening, which can be unstable during GAN training, the authors utilize Cholesky decomposition. This method is computationally efficient, differentiable, and avoids the gradient explosion issues common in adversarial settings.
- Coloring: To restore the network's representation capacity after whitening, a "coloring" step is introduced. Instead of the scalar scaling and shifting used in BN, coloring employs learnable $1 \times 1$ convolutions to project whitened features into a rich multivariate Gaussian distribution.
For conditional GANs (cGANs), the authors extend this to Conditional Coloring (cWC), where coloring filters are class-specific. To handle datasets with many classes, a Soft-Assignment ($cWC_{sa}$) mechanism is used, blending a shared dictionary of filters with class-specific weights to maintain parameter efficiency.
Experimental Setup
The WC transform was evaluated across several standard benchmarks:
- CIFAR-10 & CIFAR-100
- STL-10 (48x48)
- ImageNet & Tiny ImageNet (64x64)
- MNIST & Fashion-MNIST
Key Insights
- Synergy of Whitening and Coloring: Ablation studies show that whitening alone performs poorly; the network requires the multivariate coloring step to re-introduce necessary feature correlations for high-quality image synthesis.
- Stability of Cholesky: Cholesky decomposition proved superior to ZCA for adversarial training, preventing the training collapse into "grey images" by providing better-conditioned gradients.
- Adversarial Specificity: While WC provides marginal gains in standard classification tasks, its impact on GANs is transformative, suggesting that full-feature decorrelation is uniquely suited for taming the instability of min-max optimization.
Key Results
The WC transform consistently improved performance across various architectures (ResNet, DCGAN) and training protocols (WGAN-GP, Spectral Normalization).
- State-of-the-Art Performance: On conditional CIFAR-10, cWC achieved an Inception Score (IS) of 9.06, surpassing previous benchmarks.
- Accelerated Convergence: WC-equipped networks demonstrate massive speedups, reaching higher IS values in 20,000 iterations than baseline models achieve in 50,000.
- Efficiency: The soft-assignment variant ($cWC_{sa}$) on ImageNet achieved superior quality with only ~13% of the parameters compared to standard baseline models.
Conclusion
While the WC transform introduces a slight computational overhead (approximately 1.32x per iteration), this is largely offset by faster convergence and significantly improved image quality. Future work may explore the application of Cholesky-based whitening and coloring in non-visual domains and alternative architectures beyond convolutional networks.
Summary updated