Abstract
Recent years have seen adversarial losses applied to many fields, extending beyond generative modeling to conditional generative and discriminative settings. While various output activation functions and regularization approaches have been proposed, fundamental questions remain regarding their theoretical validity and empirical performance. This paper addresses two primary research questions: identifying the properties of "well-behaved" adversarial losses and evaluating how different combinations of activation functions and regularization perform in practice. By adopting a variational divergence minimization perspective, we derive necessary and sufficient conditions for a loss to behave as a divergence-like measure. Furthermore, we introduce the Discriminative Adversarial Network (DAN) framework to evaluate 168 combinations of twelve activation functions and fourteen regularization approaches. Our results indicate that no single combination dominates all settings, but specific strategies, such as Hinge losses and two-sided gradient penalties, offer superior robustness and performance.
Introduction
Since the introduction of Generative Adversarial Networks (GANs), the landscape of adversarial learning has been populated by a diverse array of loss functions, including Minimax, Wasserstein, and Least Squares. Despite their success, the field has lacked a unified theoretical framework to explain why certain activation functions converge while others fail. Moreover, the entanglement of loss functions with regularization "tricks"—such as Gradient Penalties or Spectral Normalization—has made it difficult to isolate the true drivers of performance. This study provides a rigorous roadmap to decouple these effects and establish a theoretical foundation for future adversarial loss design.
Theoretical Foundations: Defining "Well-Behaved" Losses
A central contribution of this work is the formal definition of a "well-behaved" adversarial loss. We consider a loss well-behaved if it functions as a valid divergence measure between the data distribution and the model distribution.
To analyze this, we utilize a generalized formulation for adversarial losses involving two functions, $\Psi$ and $\psi$, which characterize the landscape of the adversarial game. Our theoretical analysis culminates in Theorem 5, which establishes that a pair of activation functions is well-behaved if and only if their sum is concave downward and they share a specific root point where their derivatives are opposite. This derivation reveals a large class of theoretically valid losses, including two newly proposed variants:
- Absolute Loss: A piecewise linear loss function.
- Asymmetric Loss: A loss demonstrating that the functions for real and fake data do not need to be symmetric to remain theoretically sound.
Empirical Methodology: The DAN Framework
To bridge the gap between theory and practice, we developed the Discriminative Adversarial Network (DAN) framework. In a DAN, the "Generator" acts as a classifier (e.g., for MNIST digits), while the "Discriminator" attempts to distinguish between real (image, true label) pairs and fake (image, predicted label) pairs. This setup allows for the use of classification accuracy as a stable, standard metric for evaluation, avoiding the instability and subjectivity often associated with generative metrics like Inception Score or FID.
Using this framework, we conducted a massive benchmark of 168 combinations, testing 12 output activations against 14 regularization approaches on the MNIST dataset. This scale allows us to observe patterns that smaller studies might miss.
The Role of Regularization and Stability
One of the most significant findings of our empirical study is that the choice of regularization often outweighs the choice of the activation function itself in terms of impact on stability and final performance.
Our analysis of gradient penalties revealed that two-sided penalties—which penalize gradients for being either too large or too small—consistently outperform one-sided versions. Specifically, the Two-sided Local Gradient Penalty (TLGP) emerged as a top performer. Furthermore, we found that Spectral Normalization (SN) acts as a critical stabilizer, improving the robustness of all gradient penalties and preventing training from diverging prematurely.
Key Insights and Results
- The Dominance of Hinge Loss: The Hinge loss, traditionally associated with Support Vector Machines, consistently outperformed the "classic" GAN minimax loss across nearly all regularization settings.
- Asymmetry is Effective: The proposed Asymmetric loss proved to be the most robust activation function in the study, validating our theoretical finding that mirrored functions are not a requirement for success.
- The Winning Combination: The global lowest error rate in our benchmark was achieved by combining the Relativistic Hinge loss with the TLGP.
- Optimizer Dynamics: Contrary to common GAN heuristics that suggest low momentum, we found that for two-sided penalties, using a larger momentum in the Adam optimizer actually yielded better results.
Conclusion
This research provides both a theoretical sieve for valid adversarial losses and an empirical guide for their application. While the DAN framework focuses on classification, the underlying adversarial mechanisms are generic, suggesting these insights are broadly applicable to generative modeling. Future work will explore the neighborhoods around saddle points in the loss landscape to further understand the early-stage stability of these functions. Our findings serve as a reference for researchers to move beyond trial-and-error when designing adversarial systems.
Summary updated