Simple Yet Effective Way for Improving the Performance of GAN

Yoon-Jae Yeo, Yong-Goo Shin, Seung Park, Sung-Jea Ko

Abstract

In adversarial learning, the discriminator often fails to guide the generator successfully because it distinguishes between real and generated images using non-robust or "silly" features. This paper presents a simple yet effective way to improve Generative Adversarial Network (GAN) performance without imposing training overhead or modifying existing network architectures. The proposed method employs a novel Cascading Rejection (CR) module for the discriminator, which iteratively extracts multiple non-overlapping features using a vector rejection operation. By preventing the discriminator from concentrating on a single set of features, the CR module guides the generator to produce images more similar to the real data distribution. Quantitative evaluations across datasets like CIFAR-10, CelebA, and LSUN show significant improvements in Frechet Inception Distance (FID), confirming enhanced image diversity and quality.

The Problem: Discriminator "Laziness"

Generative Adversarial Networks (GANs) are notoriously difficult to train. A common failure mode is that the discriminator becomes "lazy." In a standard GAN, the final layer typically performs an inner product between a weight vector and an image feature vector to produce a single score.

Mathematically, this means the discriminator only focuses on the part of the feature space parallel to that weight vector, completely ignoring the "perpendicular" feature space. If the discriminator identifies a non-robust feature (like a specific pixel artifact) that distinguishes real from fake, it stops learning. Consequently, the generator is never pressured to learn the complex, high-dimensional details of the actual data distribution, leading to poor image quality or "mode collapse."

The discriminator typically ignores features perpendicular to the weight vector W, focusing only on the projection Vp.

The Cascading Rejection (CR) Module

To force the discriminator to be more thorough, the authors propose the Cascading Rejection (CR) module. This module acts as a lightweight "plug-and-play" addition that forces the discriminator to look at multiple, non-overlapping feature sets.

  1. Vector Rejection: Instead of producing one scalar value, the CR module works iteratively. After the first inner product, it uses a vector rejection operation to subtract the part of the feature vector that the discriminator just used.
  2. Iterative Extraction: This "leftover" feature vector is then passed to another inner product with a different weight vector. This process repeats $N$ times.
  3. Non-Overlapping Features: Because of the rejection step, each subsequent scalar value is derived from a part of the feature space that is orthogonal (perpendicular) to the previous ones.
  4. Modified Loss: The final adversarial loss is calculated by averaging the results of these $N$ different "views" of the image.
The CR module iteratively extracts scalar values by applying vector rejection to the feature vector V before each subsequent inner product.

This method requires only a few basic vector operations at the end of the discriminator, adding virtually zero computational cost and requiring no changes to the rest of the network's architecture.

Key Insights and Results

By forcing the discriminator to look at $N$ different directions in the feature space, the generator can no longer "cheat" by fixing one specific artifact. It must improve the image across all $N$ feature dimensions to fool the discriminator. This orthogonality ensures that each "head" of the CR module is looking at a unique, non-overlapping part of the data.

Quantitative Performance

The primary metric used was the Frechet Inception Distance (FID), where a lower score indicates better quality and diversity. Key results include:

  • CIFAR-10: FID improved from 16.02 (traditional GAN) to 13.90 ($N=16$).
  • LSUN: FID dropped from 19.39 to 16.77.
  • High-Resolution Success: On CelebA-HQ (512x512), the CR module improved the FID from 32.63 to 30.82.
  • Zero Overhead: Training time per iteration remained almost identical to the baseline, making it a "free" upgrade for most GAN pipelines.

Conclusion

The Cascading Rejection module provides a simple, architecture-agnostic way to stabilize GAN training and improve image synthesis. While the method currently focuses on the final layer of the discriminator, its success suggests that the principle of cascading rejection could be applied to other areas of deep learning where diverse feature extraction is beneficial, such as self-supervised learning.

Summary updated

Abstract

In adversarial learning, the discriminator often fails to guide the generator successfully since it distinguishes between real and generated images using silly or nonrobust features. To alleviate this problem, this brief presents a simple but effective way that improves the performance of the generative adversarial network (GAN) without imposing the training overhead or modifying the network architectures of existing methods. The proposed method employs a novel cascading rejection (CR) module for discriminator, which extracts multiple nonoverlapped features in an iterative manner using the vector rejection operation. Since the extracted diverse features prevent the discriminator from concentrating on nonmeaningful features, the discriminator can guide the generator effectively to produce images that are more similar to the real images. In addition, since the proposed CR module requires only a few simple vector operations, it can be readily applied to existing frameworks with marginal training overheads. Quantitative evaluations on various data sets, including CIFAR-10, CelebA, CelebA-HQ, LSUN, and tiny-ImageNet, confirm that the proposed method significantly improves the performance of GAN and conditional GAN in terms of the Frechet inception distance (FID), indicating the diversity and visual appearance of the generated images.
BibTeX
@article{yeo2021simple,
  author = {Yoon-Jae Yeo and Yong-Goo Shin and Seung Park and Sung-Jea Ko},
  title = {{Simple Yet Effective Way for Improving the Performance of GAN}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2021},
  doi = {10.1109/tnnls.2020.3045000},
}
Figure