Towards a Deeper Understanding of Adversarial Losses

Hao-Wen Dong, Yi-Hsuan YangarXiv 2019 · OpenReview

Abstract

Recent years have seen adversarial losses applied to many fields, extending beyond generative modeling to conditional generative and discriminative settings. While various output activation functions and regularization approaches have been proposed, fundamental questions remain regarding their theoretical validity and empirical performance. This paper addresses two primary research questions: identifying the properties of "well-behaved" adversarial losses and evaluating how different combinations of activation functions and regularization perform in practice. By adopting a variational divergence minimization perspective, we derive necessary and sufficient conditions for a loss to behave as a divergence-like measure. Furthermore, we introduce the Discriminative Adversarial Network (DAN) framework to evaluate 168 combinations of twelve activation functions and fourteen regularization approaches. Our results indicate that no single combination dominates all settings, but specific strategies, such as Hinge losses and two-sided gradient penalties, offer superior robustness and performance.

Introduction

Since the introduction of Generative Adversarial Networks (GANs), the landscape of adversarial learning has been populated by a diverse array of loss functions, including Minimax, Wasserstein, and Least Squares. Despite their success, the field has lacked a unified theoretical framework to explain why certain activation functions converge while others fail. Moreover, the entanglement of loss functions with regularization "tricks"—such as Gradient Penalties or Spectral Normalization—has made it difficult to isolate the true drivers of performance. This study provides a rigorous roadmap to decouple these effects and establish a theoretical foundation for future adversarial loss design.

Theoretical Foundations: Defining "Well-Behaved" Losses

A central contribution of this work is the formal definition of a "well-behaved" adversarial loss. We consider a loss well-behaved if it functions as a valid divergence measure between the data distribution and the model distribution.

To analyze this, we utilize a generalized formulation for adversarial losses involving two functions, $\Psi$ and $\psi$, which characterize the landscape of the adversarial game. Our theoretical analysis culminates in Theorem 5, which establishes that a pair of activation functions is well-behaved if and only if their sum is concave downward and they share a specific root point where their derivatives are opposite. This derivation reveals a large class of theoretically valid losses, including two newly proposed variants:

  • Absolute Loss: A piecewise linear loss function.
  • Asymmetric Loss: A loss demonstrating that the functions for real and fake data do not need to be symmetric to remain theoretically sound.

Empirical Methodology: The DAN Framework

To bridge the gap between theory and practice, we developed the Discriminative Adversarial Network (DAN) framework. In a DAN, the "Generator" acts as a classifier (e.g., for MNIST digits), while the "Discriminator" attempts to distinguish between real (image, true label) pairs and fake (image, predicted label) pairs. This setup allows for the use of classification accuracy as a stable, standard metric for evaluation, avoiding the instability and subjectivity often associated with generative metrics like Inception Score or FID.

Using this framework, we conducted a massive benchmark of 168 combinations, testing 12 output activations against 14 regularization approaches on the MNIST dataset. This scale allows us to observe patterns that smaller studies might miss.

The Role of Regularization and Stability

One of the most significant findings of our empirical study is that the choice of regularization often outweighs the choice of the activation function itself in terms of impact on stability and final performance.

Comparison of different gradient penalty strategies evaluated in the study, including coupled, local, R1, and R2 approaches.

Our analysis of gradient penalties revealed that two-sided penalties—which penalize gradients for being either too large or too small—consistently outperform one-sided versions. Specifically, the Two-sided Local Gradient Penalty (TLGP) emerged as a top performer. Furthermore, we found that Spectral Normalization (SN) acts as a critical stabilizer, improving the robustness of all gradient penalties and preventing training from diverging prematurely.

Key Insights and Results

  • The Dominance of Hinge Loss: The Hinge loss, traditionally associated with Support Vector Machines, consistently outperformed the "classic" GAN minimax loss across nearly all regularization settings.
  • Asymmetry is Effective: The proposed Asymmetric loss proved to be the most robust activation function in the study, validating our theoretical finding that mirrored functions are not a requirement for success.
  • The Winning Combination: The global lowest error rate in our benchmark was achieved by combining the Relativistic Hinge loss with the TLGP.
  • Optimizer Dynamics: Contrary to common GAN heuristics that suggest low momentum, we found that for two-sided penalties, using a larger momentum in the Adam optimizer actually yielded better results.

Conclusion

This research provides both a theoretical sieve for valid adversarial losses and an empirical guide for their application. While the DAN framework focuses on classification, the underlying adversarial mechanisms are generic, suggesting these insights are broadly applicable to generative modeling. Future work will explore the neighborhoods around saddle points in the loss landscape to further understand the early-stage stability of these functions. Our findings serve as a reference for researchers to move beyond trial-and-error when designing adversarial systems.

Summary updated

Abstract

Recent years have seen adversarial losses been applied to many fields. Their applications extend beyond the originally proposed generative modeling to conditional generative and discriminative settings. While prior work has proposed various output activation functions and regularization approaches, some open questions still remain unanswered. In this paper, we aim to study the following two research questions: 1) What types of output activation functions form a well-behaved adversarial loss? 2) How different combinations of output activation functions and regularization approaches perform empirically against one another? To answer the first question, we adopt the perspective of variational divergence minimization and consider an adversarial loss well-behaved if it behaves as a divergence-like measure between the data and model distributions. Using a generalized formulation for adversarial losses, we derive the necessary and sufficient conditions of a well-behaved adversarial loss. Our analysis reveals a large class of theoretically valid adversarial losses. For the second question, we propose a simple comparative framework for adversarial losses using discriminative adversarial networks. The proposed framework allows us to efficiently evaluate adversarial losses using a standard evaluation metric such as the classification accuracy. With the proposed framework, we evaluate a comprehensive set of 168 combinations of twelve output activation functions and fourteen regularization approaches on the handwritten digit classification problem to decouple their effects. Our empirical findings suggest that there is no single winning combination of output activation functions and regularization approaches across all settings. Our theoretical and empirical results may together serve as a reference for choosing or designing adversarial losses in future research.
BibTeX
@misc{dong2019towards,
  author = {Hao-Wen Dong and Yi-Hsuan Yang},
  title = {{Towards a Deeper Understanding of Adversarial Losses}},
  year = {2019},
  eprint = {1901.08753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.08753},
}
Figure