Generative Adversarial Networks (GANs) are powerful tools for modeling complex data distributions, but they are notoriously difficult to train. They frequently suffer from unstable convergence—where the generator and discriminator fail to reach an equilibrium—and mode collapse, where the generator produces only a limited variety of outputs.
The core of a GAN is the divergence (or "distance") it tries to minimize between the real data distribution and the generated one. While Vanilla GANs use the Jensen-Shannon divergence and others like WGAN use the Wasserstein distance, researchers are constantly searching for better mathematical measures. The Hockey-Stick GAN (HS-GAN) introduces a new objective function based on the "hockey-stick divergence," a flexible member of the $f$-divergence family that offers a potentially more stable training process.
The Hockey-Stick Objective
The authors leverage the $f$-GAN framework, which allows any $f$-divergence to be transformed into a GAN objective using a variational lower bound.
They utilize the hockey-stick function: $f_\gamma(t) = \max(t - \gamma, 0)$ for $\gamma \ge 1$. Visually, this function resembles a hockey stick—it remains flat until it hits a threshold ($\gamma$), after which it rises linearly.
This leads to a surprisingly simple minimax objective: $$\min_G \max_{D: \mathcal{X} \to [0,1]} \mathbb{E}_{x \sim \mathbb{P}_r}[D(x)] - \gamma \mathbb{E}_{x \sim \mathbb{P}_g}[D(x)]$$
In this setup, the discriminator $D$ outputs a probability between 0 and 1. The generator $G$ tries to minimize the difference between the expected discriminator scores on real and generated data, while the discriminator tries to maximize it.
The key to this approach is the hyperparameter $\gamma$. When $\gamma = 1$, the objective is equivalent to the Total Variation distance. As $\gamma$ increases, the divergence becomes more "forgiving," ignoring certain differences between the real and generated distributions.
Key Insights
- Information Richness: The set of hockey-stick divergences for all $\gamma > 0$ contains enough information to uniquely determine almost any other $f$-divergence (such as KL or Hellinger). This makes it a "universal" building block for measuring distribution differences.
- The Identity Trade-off: Unlike most divergences, for $\gamma > 1$, the hockey-stick divergence can be zero even if the two distributions are not identical. While this sounds counterintuitive, this flexibility might actually help the generator avoid getting stuck in bad local optima during the early stages of training.
- Simplicity: The resulting objective function is linear with respect to the discriminator's outputs, making it computationally straightforward to implement compared to more complex divergence measures.
Experimental Results: Fitting a Gaussian
To test the HS-GAN, the authors focused on a proof-of-concept evaluation using synthetic 1D Gaussian data (mean $\mu = 0$, variance $\sigma^2 = 2$). They deliberately used low-capacity neural networks to observe how different objectives handle limited model power.
In this low-capacity setting, the Vanilla GAN failed to accurately fit the target Gaussian distribution. Similarly, the HS-GAN with $\gamma = 1$ (equivalent to Total Variation distance) also struggled, producing a skewed distribution that failed to align with the target curve.
However, the power of the $\gamma$ parameter becomes evident when tuned. While $\gamma = 0.5$ failed to align with the target, setting $\gamma = 2$ allowed the HS-GAN to successfully fit the real distribution. This demonstrates that adjusting $\gamma$ can help the model converge in scenarios where standard methods fail.
Limitations and Future Work
While promising, the HS-GAN approach has a few notable limitations:
- Lack of Identity Guarantee: Because the divergence can be zero even when distributions differ (for $\gamma > 1$), there is no strict theoretical guarantee that the generator will converge to the exact true data distribution without careful tuning.
- Scale of Experiments: The current results are limited to 1D synthetic data. It remains to be seen how HS-GAN performs on high-dimensional, complex datasets like images (e.g., CIFAR-10 or ImageNet).
- Hyperparameter Overhead: Introducing $\gamma$ adds another hyperparameter to the already tricky GAN tuning process. The authors suggest that a schedule for varying $\gamma$ during training could be highly beneficial, leaving this as an open area for future research.
Summary updated