Abstract
Distributional Reinforcement Learning (RL) aims to model the complete probability distribution of future returns rather than just their expected value. We present GAN Q-learning, a novel approach that leverages the implicit modeling power of Generative Adversarial Networks (GANs) to represent these reward distributions. By utilizing a Wasserstein GAN (WGAN) framework, the algorithm minimizes the Earth-Mover distance between predicted and target distributions, providing a flexible, "black-box" alternative to traditional methods that require predefined probability supports.
Introduction
Standard RL typically estimates the mean of future returns, which fails to capture the inherent variability or risk associated with specific states. Distributional RL has emerged as a robust alternative, improving performance and stability in complex tasks by learning the entire return distribution. However, existing methods often rely on explicit representations, such as fixed categorical bins (e.g., the C51 algorithm), which can be computationally restrictive. GAN Q-learning addresses this by using GANs to learn the shape of reward distributions organically through an adversarial game.
Methodology
The GAN Q-learning framework adapts the Wasserstein GAN with Gradient Penalty (WGAN-GP) to the RL setting:
- Generator ($G$): Takes a state and a noise vector ($z$) as input to produce a sample from the return distribution for each possible action.
- Discriminator ($D$): Acts as a critic, attempting to distinguish between samples from the generator and "real" samples derived from the Distributional Bellman Update (immediate reward plus the discounted sample from the subsequent state).
- Optimization: The model minimizes the Wasserstein-1 (Earth-Mover) distance without requiring an explicit definition of the reward support.
- Implicit Exploration: Early in training, the generator produces overlapping distributions for different actions, facilitating natural exploration as the agent samples various "optimal" actions before the network clearly separates their values.
Experimental Evaluation
The algorithm was evaluated across several environments to verify its mathematical foundations and benchmark its performance:
- Tabular Environments: A 10-state chain (Two Goal Chain), a 6x6 Gridworld, and a Two-state MDP.
- Continuous Benchmarks: OpenAI Gym's Cartpole-v0 and Acrobot-v1.
Results and Discussion
GAN Q-learning (referred to as GAN-DQN) demonstrated performance parity with traditional Q-learning and other distributional baselines in tabular environments. In the Cartpole-v0 task, it successfully learned optimal policies and, in certain configurations, outperformed standard DQN.
Key insights from the study include:
- Distributional Accuracy: Visualizations confirm the model successfully learns to separate reward distributions for "good" versus "bad" actions over time, as seen in the transition from Episode 1 to Episode 15 in the provided figures.
- Exploration as a Side Effect: The inherent "fuzziness" of the initial generator distributions acts as a built-in exploration mechanism, potentially reducing the reliance on traditional $\epsilon$-greedy strategies.
- Hyperparameter Sensitivity: While WGAN-GP typically uses a strong gradient penalty ($\lambda=10$), the authors found that a weaker penalty ($\lambda=0.1$) performed better in the volatile environment of RL.
Conclusion and Future Work
GAN Q-learning offers a powerful, flexible framework for distributional RL by treating reward modeling as a black-box adversarial task. While it achieves performance parity with existing methods, GAN training remains sensitive to sparse rewards and requires careful stabilization, such as the use of target networks. Future research will focus on scaling the architecture with Convolutional Neural Networks (CNNs) for visual inputs and exploring integration with more complex policy gradient algorithms.
Summary updated