A Characteristic Function Approach to Deep Implicit Generative Modeling.

Abdul Fatir Ansari, Jonathan Scarlett, Harold Soh

Abstract

Implicit Generative Models (IGMs) such as GANs have emerged as effective data-driven models for generating samples, particularly images. In this paper, we formulate the problem of learning an IGM as minimizing the expected distance between characteristic functions. Specifically, we minimize the distance between characteristic functions of the real and generated data distributions under a suitably-chosen weighting distribution. This distance metric, which we term as the characteristic function distance (CFD), can be (approximately) computed with linear time-complexity in the number of samples, in contrast with the quadratic-time Maximum Mean Discrepancy (MMD). By replacing the discrepancy measure in the critic of a GAN with the CFD, we obtain a model that is simple to implement and stable to train. The proposed metric enjoys desirable theoretical properties including continuity and differentiability with respect to generator parameters, and continuity in the weak topology. We further propose a variation of the CFD in which the weighting distribution parameters are also optimized during training; this obviates the need for manual tuning, and leads to an improvement in test power relative to CFD. We demonstrate experimentally that our proposed method outperforms WGAN and MMD-GAN variants on a variety of unsupervised image generation benchmarks.

Introduction

Training Implicit Generative Models (IGMs) like GANs is notoriously difficult. The original GAN formulation uses Jensen-Shannon divergence, which often suffers from "vanishing gradients"—where the critic becomes too good too quickly, providing no useful signal to the generator.

To address this, researchers turned to Integral Probability Metrics (IPMs) like the Wasserstein distance (WGAN) and Maximum Mean Discrepancy (MMD-GAN). While effective, MMD has a major drawback: its computational cost grows quadratically ($O(n^2)$) with the number of samples, making it expensive for large batches. This work introduces the Characteristic Function Distance (CFD), a metric that combines the theoretical advantages of IPMs (continuity and differentiability) with the computational efficiency of a linear-time algorithm.

Methodology: The Characteristic Function Distance

The core idea is to compare distributions in the frequency domain using Characteristic Functions (CFs). In probability theory, every distribution has a unique CF (the Fourier transform of its density). If two distributions have the same CF, they are identical.

The CFD Metric and Empirical Approximation

The CFD is defined as the weighted integral of the squared difference between the CFs of the real and generated distributions. In practice, this integral is approximated by sampling a finite set of $k$ random frequencies from a weighting distribution (e.g., Gaussian or Student-t). This Empirical CFD (ECFD) reduces complexity to $O(nk)$, making it linear in the number of samples $n$.

Optimized CFD (OCF-GAN)

A key innovation is the optimization of the weighting distribution parameters (such as the scale $\sigma$) during training. This allows the "critic" to adaptively find the frequencies that best distinguish real data from fake data. As shown in Figure 1, optimizing these parameters significantly improves the test power of the metric, especially as the dimensionality of the data increases.

Test power comparison across dimensions. Optimized CFD (OECFD) maintains higher power in high-dimensional spaces compared to fixed ECFD.

Experimental Results

The proposed OCF-GAN was evaluated across several benchmarks, including synthetic 1D data, MNIST, CIFAR10, STL10, and CelebA.

Convergence and Accuracy

In synthetic tests, OCF-GAN variants reached the ground truth distribution much faster than MMD-based models. Figure 2 illustrates the Mean Absolute Error (MAE) over generator iterations, highlighting the superior convergence speed and stability of OCF-GAN-GP compared to WGAN and MMD-GAN.

Convergence comparison on synthetic data. OCF-GAN-GP variants demonstrate faster convergence and lower Mean Absolute Error (MAE) compared to WGAN and MMD-GAN.

Image Generation Performance

On the CelebA dataset, OCF-GAN-GP (using Gaussian weighting and gradient penalty) achieved an FID score of 9.39, outperforming WGAN-GP (10.03) and MMD-GAN-GP (13.22). The model successfully scaled to $128 \times 128$ images using a ResNet architecture, producing sharp, high-quality celebrity faces competitive with more complex architectures.

Discussion and Conclusion

The CFD approach offers a powerful alternative to existing GAN metrics by leveraging the Fourier domain. Its linear-time complexity makes it highly scalable, while the adaptive "critic" (via optimized weighting) ensures robust training. One of the most surprising findings is that OCF-GAN performs exceptionally well even with a very small number of random frequencies ($k=8$ or even $k=1$), suggesting that a few well-chosen "views" of the distribution are sufficient for training.

Future work may explore more sophisticated frequency sampling techniques to reduce gradient variance and investigate the specific geometry of the CFD loss function to develop specialized regularizers. Beyond image generation, CFD holds promise for other distribution-matching tasks such as domain adaptation and separation.

Summary updated

Abstract

Implicit Generative Models (IGMs) such as GANs have emerged as effective data-driven models for generating samples, particularly images. In this paper, we formulate the problem of learning an IGM as minimizing the expected distance between characteristic functions. Specifically, we minimize the distance between characteristic functions of the real and generated data distributions under a suitably-chosen weighting distribution. This distance metric, which we term as the characteristic function distance (CFD), can be (approximately) computed with linear time-complexity in the number of samples, in contrast with the quadratic-time Maximum Mean Discrepancy (MMD). By replacing the discrepancy measure in the critic of a GAN with the CFD, we obtain a model that is simple to implement and stable to train. The proposed metric enjoys desirable theoretical properties including continuity and differentiability with respect to generator parameters, and continuity in the weak topology. We further propose a variation of the CFD in which the weighting distribution parameters are also optimized during training; this obviates the need for manual tuning, and leads to an improvement in test power relative to CFD. We demonstrate experimentally that our proposed method outperforms WGAN and MMD-GAN variants on a variety of unsupervised image generation benchmarks.
BibTeX
@inproceedings{ansari2020characteristic,
  author = {Abdul Fatir Ansari and Jonathan Scarlett and Harold Soh},
  title = {{A Characteristic Function Approach to Deep Implicit Generative Modeling.}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=BAxbfxjkDe9},
}
Figure