Neural Characteristic Function Learning for Conditional Image Generation

Shengxi Li, Jialu Zhang, Yifei Li, Mai Xu, Xin DengICCV 2023 · DOI

Abstract

Conditional Generative Adversarial Networks (cGANs) often suffer from training instabilities and mode collapse due to the limitations of Jensen-Shannon divergence in high-dimensional spaces. This paper introduces CCF-GAN, a framework that utilizes Neural Characteristic Functions (NCFs) to measure distributional distances. By explicitly decomposing joint distributions and learning optimal projections, CCF-GAN achieves state-of-the-art performance and remarkable stability across various benchmarks, including ImageNet and VGGFace2.

Introduction

Conditional image generation aims to produce high-fidelity images based on specific inputs like class labels. Traditional cGANs typically rely on the Jensen-Shannon (JS) divergence, implemented via cross-entropy loss, to distinguish real from synthetic data. However, JS divergence is a "bin-to-bin" comparison that often leads to vanishing gradients when distributions do not overlap, causing the discriminator to overpower the generator early in training. While Integral Probability Metrics (IPMs) like Wasserstein distance offer improvements, they often struggle with the complex coupling between high-dimensional image data and low-dimensional labels.

Methodology: Neural Characteristic Functions

The authors propose moving from probability density functions to Characteristic Functions (CFs). Every probability distribution possesses a unique CF that completely defines it. Comparing distributions via CFs provides smoother gradients and a theoretically "complete" comparison.

To overcome the "curse of dimensionality" associated with calculating CFs for high-resolution images, the paper introduces the Neural CF (NCF). Instead of using fixed frequency sampling, an adversarial network learns optimal projections to maximize the detected difference between real and generated distributions. Furthermore, CCF-GAN explicitly decomposes the joint distribution $p(x, y)$ into marginal image and conditional label distributions, allowing for more nuanced feature processing.

Architectural comparison of OCFGAN, RCFGAN, and the proposed CCF-GAN, illustrating the evolution of distributions on the complex plane during training.

Experimental Results and Stability

CCF-GAN was evaluated on synthetic 2D data, CIFAR10, VGGFace2, and ImageNet (128x128). The results demonstrate a significant leap over existing state-of-the-art (SOTA) methods:

  • ImageNet (128x128): CCF-GAN achieved an FID of 11.34, nearly halving the FID of BigGAN (22.77).
  • Inception Score (IS): On ImageNet, it reached an IS of 180.84, nearly tripling the performance of ReACGAN (68.27).
  • Stability: In tests across 48 challenging training conditions (varying learning rates and batch sizes), CCF-GAN successfully trained in almost every scenario, whereas competitors like BigGAN and ACGAN failed in over 50% of the trials.
Training loss and generated MNIST-like samples over epochs, highlighting the stability and convergence of the model with and without labels.

Key Insights

  • Global Signatures: CFs capture global properties of distributions, making them naturally resistant to the misalignment issues that plague standard GANs.
  • Adversarial Frequency Learning: The NCF acts as an intelligent critic, learning which features are most effective for distinguishing real data from fakes.
  • Theoretical Soundness: The authors provide proofs that the NCF provides a valid distance metric and that minimizing NCF loss effectively reduces the true difference between distributions.

Conclusion and Future Work

CCF-GAN represents a significant advancement in stable conditional image generation. While the model introduces a hyperparameter $k$ for projection samples and adds some computational complexity to the discriminator, its superior stability and performance on high-resolution benchmarks make it a robust alternative to traditional cross-entropy-based GANs. Future work may explore scaling these techniques to ultra-high resolutions (e.g., 1024x1024) and further optimizing the NCF training time.

Summary updated

BibTeX
@inproceedings{li2023neural,
  author = {Shengxi Li and Jialu Zhang and Yifei Li and Mai Xu and Xin Deng and Li Li},
  title = {{Neural Characteristic Function Learning for Conditional Image Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00662},
}
Figure