In the world of Generative Adversarial Networks (GANs), researchers have spent years perfecting the "cat-and-mouse" game between a Generator trying to create realistic images and a Discriminator trying to spot the fakes. While sophisticated metrics like SSIM or NIQE are frequently used to evaluate how well these models perform after training, they are rarely used to actually guide the training process itself.
In their NeurIPS 2019 paper, "Quality Aware Generative Adversarial Networks" (QAGANs), Parimala Kancharla and Sumohana S. Channappayya bridge this gap. They demonstrate that by baking human-centric image quality metrics directly into the GAN loss function, models can achieve significantly better visual results and more stable training.
The Disconnect Between Training and Evaluation
Despite their immense popularity, GANs are notoriously difficult to train. They frequently suffer from mode collapse (where the generator produces the same few images), convergence issues, and "noisy" visual outputs that lack the crispness of real photos.
Researchers typically tackle these problems using mathematical constraints, such as the Lipschitz continuity enforced in Wasserstein GANs (WGAN-GP). However, these constraints are often purely mathematical and do not explicitly account for what makes an image look "good" to the human eye. The authors identified a crucial gap: while we have excellent tools to measure image quality, they are rarely used as cost functions because they are mathematically unwieldy—often non-convex and difficult to differentiate.
Making GANs "Quality Aware"
To address this, the authors propose two primary methods to integrate objective Image Quality Assessment (IQA) metrics as regularizers within the GAN framework:
1. SSIM-based Regularization
The Structural Similarity Index (SSIM) is a gold standard for comparing a distorted image to a perfect reference image. Because standard SSIM is not a valid mathematical distance metric (it can yield negative values), the authors utilized a specific variant called $d^Q$. This transformation turns SSIM into a valid distance metric, which is then used to enforce a "Quality Aware" Lipschitz constraint on the discriminator.
2. NIQE-based Gradient Penalty
The Natural Image Quality Evaluator (NIQE) is a "no-reference" metric, meaning it can judge an image's quality without needing a perfect baseline. It evaluates the statistical signature of an image, specifically looking at Mean Subtracted Contrast Normalized (MSCN) coefficients.
The authors made a fascinating discovery: the gradients of the discriminator also follow these natural statistical signatures. They proposed a novel penalty function that punishes the discriminator if its gradients deviate from the statistical "fingerprint" of real-world natural images.
These regularizers were integrated into the WGAN-GP framework, creating a hybrid loss function that balances traditional adversarial training with explicit quality constraints.
Key Insights
- The Statistical Signature of Gradients: Perhaps the most surprising finding is that the discriminator's gradients are not just random noise. When trained on natural images, they inherit the same Gaussian-like statistical properties as the images themselves.
- Bridging the Gap: By using IQA metrics during training, the model essentially "pre-tests" itself against the very metrics researchers use to judge it at the end.
- Stability through Structure: Traditional gradient penalties only care about the norm (length) of the gradient. QAGANs care about the structure and correlation of the gradient, leading to more meaningful updates for the generator.
State-of-the-Art Results
The researchers evaluated QAGANs on standard computer vision benchmarks (CIFAR-10, STL-10, and CelebA), measuring success using the Inception Score (IS) and the Frechet Inception Distance (FID).
- Massive FID Improvements: On the CIFAR-10 dataset, the standard WGAN-GP achieved an FID of 40.2. The SSIM-based QAGAN slashed that to 13.91, and the NIQE-based version reached an impressive 12.46.
- CelebA Performance: For face generation, the SSIM-based QAGAN achieved an FID of 6.42, nearly doubling the performance of the baseline WGAN-GP (12.89).
- Scaling Up: When applied to Progressive GANs (PGGAN) at 256x256 resolution, adding QAGAN regularizers dropped the FID from 62.86 to 40.83, proving these methods scale well to high-resolution image synthesis.
Limitations and Future Work
While highly effective, the approach has a few limitations. The NIQE-based regularizer showed inconsistent performance on very small images (like the 32x32 CIFAR-10 dataset), likely due to the difficulty of estimating statistical parameters from so few pixels. Additionally, calculating MSCN coefficients and Multivariate Gaussian distances adds computational complexity to the training loop.
However, this work paves the way for a new paradigm in generative modeling. The authors suggest that the plethora of excellent quality assessment metrics available in the image processing community could soon be translated into even more powerful GAN regularizers.
Summary updated