Elastic Interaction Energy-Based Generative Model: Approximation in Feature Space

Chuqi Chen, Yue Wu, Yang XiangarXiv 2023 · DOI · arXiv

Generative Adversarial Networks (GANs) are powerful tools for data generation but are frequently hampered by training instability and mode collapse. A novel approach, the Elastic Interaction Energy-Based Generative (EIEG) model, addresses these issues by drawing inspiration from the physics of crystal defects. By modeling data points as particles interacting via elastic forces in a latent feature space, the EIEG GAN achieves enhanced stability and superior distribution coverage.

Addressing the Challenges of Generative Modeling

The development of EIEG GAN was motivated by two persistent problems in the field:

  1. Mode Collapse: Standard GAN metrics, such as Jensen-Shannon or Wasserstein distance, often fail to incentivize the generator to cover the entire target distribution. This leads to the generator producing a limited variety of samples.
  2. Data Sparsity: High-dimensional data is inherently sparse, making it difficult to compute meaningful distances between distributions directly in the pixel space.

The EIEG Framework

The core of the proposed model is the Elastic Interaction Energy (EIE) loss function. In physics, defects in a crystal lattice interact through long-range elastic forces. The EIEG model adapts this principle by defining a loss with two components:

  • Attractive Force: This component pulls generated samples toward the manifold of real data.
  • Repulsive Self-Interaction: This component pushes generated samples away from one another, ensuring "mutual exclusion" and forcing the generator to explore and occupy the entire distribution, thereby preventing mode collapse.
The EIEG GAN architecture utilizes an Elastic Discriminator as a feature transformer to map data into a compact latent space where EIE loss is calculated.

As shown in the architecture diagram, the model employs an Elastic Discriminator ($D_\phi$). Unlike traditional discriminators that output a binary classification, $D_\phi$ acts as a feature transformer. It maps high-dimensional, scattered input data into a lower-dimensional latent feature space. The EIE loss is then computed within this more manageable space, allowing the physics-based regularizer to operate effectively.

Mathematical Stability

To ensure robust training, the authors introduced a stabilizing term derived from Wasserstein gradient flow analysis. By applying Fourier transforms and partial differential equations (PDEs), they demonstrated that this term prevents the training process from becoming unstable, a common issue when using long-range potentials.

Experimental Results

The EIEG GAN was evaluated on synthetic datasets, such as the 25-Gaussians task, and real-world image benchmarks including MNIST, CIFAR-10, and CelebA.

Solving Mode Collapse

On the 25-Gaussians benchmark, the EIEG GAN successfully captured all 25 modes of the distribution. This is a significant improvement over Vanilla GANs and WGAN-GP, which often miss several clusters.

Comparison of ground truth distributions and EIEG generated results across various synthetic tasks, demonstrating effective mode capture.

Performance on Image Datasets

The model demonstrated high stability and quality on complex image datasets:

  • Stability: Training curves on FashionMNIST and CIFAR-10 were notably smoother compared to DCGAN, avoiding the sudden "crashes" in Inception Score (IS) typical of standard architectures.
  • Quantitative Metrics: On FashionMNIST, the EIEG GAN achieved an IS of 8.75, outperforming both DCGAN (8.05) and MMD-GAN.
  • Feature Space Impact: Increasing the dimensionality of the latent feature space was found to directly correlate with improved image detail, such as sharper textures and patterns.

Conclusion

The EIEG GAN represents a successful integration of physical principles into deep learning. By treating data generation as a physical interaction problem in a transformed feature space, the model overcomes the traditional pitfalls of GAN training. Future research may extend this physics-inspired loss to other domains, such as diffusion models or non-image data types like genomic sequences, where data sparsity remains a significant hurdle.

Summary updated

BibTeX
@misc{chen2023elastic,
  author = {Chuqi Chen and Yue Wu and Yang Xiang},
  title = {{Elastic Interaction Energy-Based Generative Model: Approximation in Feature Space}},
  year = {2023},
  eprint = {2303.10553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.10553},
}
Figure