Abstract: Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer. This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values. This connection allows us to estimate the Q-values from the action preferences of the policy, to which we apply Q-learning updates. We refer to the new technique as 'PGQL', for policy gradient and Q-learning. We also establish an equivalency between action-value fitting techniques and actor-critic algorithms, showing that regularized policy gradient techniques can be interpreted as advantage function learning algorithms. We conclude with some numerical examples that demonstrate improved data efficiency and stability of PGQL. In particular, we tested PGQL on the full suite of Atari games and achieved performance exceeding that of both asynchronous advantage actor-critic (A3C) and Q-learning.
Introduction
In model-free reinforcement learning, researchers typically choose between two paths: action-value fitting (e.g., Q-learning) and policy gradient methods (e.g., A3C). Q-learning methods are "off-policy," allowing them to learn from a replay buffer of past experiences, which makes them data-efficient but often unstable. Policy gradient methods are generally more stable but "on-policy," meaning they must discard data after a single update, leading to significant waste. This paper introduces PGQL (Policy Gradient and Q-Learning), a hybrid algorithm that bridges these two families to achieve both stability and efficiency.
The PGQL Method
The core of the paper is a theoretical discovery: at the fixed point of an entropy-regularized policy gradient algorithm, there is a direct functional relationship between the policy and the Q-values. Specifically, the optimal policy can be expressed as a softmax over the advantage function.
Using this insight, PGQL performs two types of updates simultaneously:
- On-policy actor-critic updates: Standard A3C-style updates to ensure stability.
- Off-policy Q-learning updates: Using data sampled from a replay buffer, the algorithm treats policy-derived Q-values as targets for optimization.
This approach allows the agent to "double-team" the optimization problem, using the replay buffer as a stabilizer that prevents the policy from changing too drastically in a single step.
Experimental Evaluation
The authors evaluated PGQL on two main environments:
- Grid World: A toy 4x6 grid used to demonstrate basic properties like data efficiency and stability under different step sizes.
- Atari 2600 Suite: The full suite of 57 games via the Arcade Learning Environment, testing the algorithm's ability to handle complex visual inputs.
Results and Discussion
In the Grid World experiments, PGQL demonstrated significantly faster convergence and higher stability than pure actor-critic or Q-learning methods. As shown in the performance plots, PGQL reaches the optimal return with far fewer agent steps.
The results on the Atari suite were even more pronounced. PGQL achieved a median human-normalized score of 145.6%, compared to 107.3% for A3C and 58.9% for Q-learning. In many games, such as Assault and Battle Zone, PGQL learned significantly faster, reaching higher scores with fewer environment interactions.
Conclusion
PGQL successfully integrates the stability of policy gradients with the data efficiency of Q-learning. By establishing entropy regularization as the mathematical "glue" between these methods, the authors provide a robust framework for off-policy learning. While the algorithm requires managing both an actor-critic setup and a replay buffer, the resulting performance gains and robustness across diverse environments represent a significant advancement in deep reinforcement learning.
Summary updated