Abstract
We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a drop-in replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in realistic adversarial games–like generative adversarial networks– by showing we can find better solutions with an almost identical computational cost. We also show a practical complex-valued Adam variant, which we use to train BigGAN to improve inception scores on CIFAR-10.
Introduction
In the landscape of deep learning, optimization is traditionally viewed as a single-player journey toward a global minimum. However, many modern architectures, such as Generative Adversarial Networks (GANs), frame optimization as a competitive game between multiple players (e.g., a generator and a discriminator). Standard momentum-based optimizers often struggle in these settings because they are designed for static, real-valued loss landscapes. In differentiable games, the Jacobian of the gradient vector field frequently exhibits imaginary eigenvalues, leading to "rotational" dynamics where players orbit each other rather than converging to a stable Nash equilibrium.
Methodology: Simultaneous Complex Momentum (SimCM)
To counteract these instabilities, we propose Simultaneous Complex Momentum (SimCM). By generalizing the momentum coefficient $\beta$ to be complex-valued ($\beta \in \mathbb{C}$), we allow the momentum buffer to oscillate. This oscillation acts as a "counter-rotation" that cancels out the natural spiraling behavior of adversarial competition.
While the internal calculations utilize complex arithmetic, the parameter updates remain real-valued by taking the real part of the complex buffer. This makes SimCM a drop-in replacement for standard optimizers in frameworks like PyTorch or JAX. We further extend this logic to create Complex Adam, where the $\beta_1$ parameter is generalized to the complex plane.
Experimental Evaluation
The effectiveness of complex momentum was tested across a variety of scenarios:
- Bilinear Zero-Sum Games: Providing a foundation for theoretical convergence proofs.
- Dirac-GAN: A simplified model used to visualize and analyze rotational dynamics.
- 2D Mixture of Gaussians: A synthetic task for evaluating generative performance.
- CIFAR-10: Training a full-scale BigGAN to demonstrate practical scalability and performance gains.
Key Results and Insights
Our experiments revealed that a "sweet spot" exists for the complex phase of $\beta$, which we term almost-positive momentum (typically $\arg(\beta) \approx \pi/8$). This setting provides the acceleration benefits of standard momentum in cooperative regions while maintaining the stability required for adversarial regions.
Key findings include:
- Convergence Stability: On the Dirac-GAN, complex momentum converges smoothly to the optimum, whereas standard momentum and SGD often diverge.
- Improved GAN Performance: Using Complex Adam to train BigGAN on CIFAR-10 resulted in an Inception Score (IS) of 9.25, outperforming the baseline of 9.10.
- Robustness: Complex momentum is less sensitive to the degree of "adversarialness" in a game compared to classical momentum and performs competitively with more complex first-order methods like Extragradient (EG).
- Efficiency: The computational overhead is minimal. While memory usage for the buffer doubles, the number of expensive gradient evaluations remains unchanged.
Discussion and Future Work
Complex momentum introduces a new hyperparameter: the phase angle of $\beta$. While $\pi/8$ is a strong default, further research could explore automated tuning or player-specific phases to account for the differing spectral properties of generators and discriminators. Additionally, fully integrating complex logic into Adam's second-moment estimation remains an open question for future optimization research.
Summary updated