Combining policy gradient and Q-learning

Brendan O'Donoghue, Remi Munos, Koray Kavukcuoglu, Volodymyr Mnih

This paper from DeepMind introduces a bridge between two historically distinct families of reinforcement learning (RL): Policy Gradients and Q-learning. By establishing a mathematical link between the two, the authors present PGQL (Policy Gradient and Q-Learning), an algorithm that combines the stability of on-policy methods with the data efficiency of off-policy replay buffers.

The Challenge in Model-Free RL

In model-free reinforcement learning, researchers typically choose between two primary paths, each with distinct trade-offs:

  1. Action-Value Fitting (e.g., Q-learning): These methods are "off-policy," meaning they can learn from a "replay buffer" of old experiences. This makes them highly data-efficient. However, they are often unstable when combined with deep neural networks.
  2. Policy Gradients (e.g., Actor-Critic): These methods represent the policy explicitly and are generally more stable. Unfortunately, vanilla versions are "on-policy"—they can only learn from data generated by the current policy. Once an action is taken and the update is performed, that data is discarded, which is incredibly wasteful.

The core challenge addressed by this paper is combining the best of both worlds: utilizing the stable gradient-based updates of policy methods while still benefiting from the massive data efficiency of a replay buffer.

The PGQL Method

The authors' primary contribution is the PGQL update. They derive a theoretical link showing that for an entropy-regularized policy (a common technique to ensure exploration), the fixed point of the policy gradient is directly related to the Q-values.

Specifically, they demonstrate that the "action preferences" (the logits before the softmax layer in a neural network) can be interpreted as scaled Q-values. This mathematical equivalence allows them to:

  • Estimate Q-values from the policy: $Q \approx \alpha \log \pi + V$.
  • Apply Q-learning updates: With a mechanism to "read" Q-values from the policy network, the agent can sample old transitions from a replay buffer and apply a standard Q-learning update (minimizing the Bellman residual) directly to the policy parameters.

The resulting algorithm, PGQL, maintains a shared neural network that outputs both a policy ($\pi$) and a state-value ($V$). It performs standard on-policy actor-critic updates in real-time while periodically sampling from a replay buffer to perform off-policy Q-learning updates on the same parameters.

Experimental Setup & Datasets

The authors evaluated PGQL on two main environments to test both theoretical properties and practical scalability:

  1. Toy Grid World: A 4x6 deterministic environment used to validate the theoretical properties and stability of the algorithm.
  2. Atari 2600 Suite: The "gold standard" benchmark for deep RL, consisting of dozens of diverse games. They compared PGQL against Asynchronous Advantage Actor-Critic (A3C) and a variant of Deep Q-Learning.
Grid world environment and performance comparison showing PGQL's faster convergence compared to Actor-critic and Q-learning.

Key Insights

  • The Regularization Link: Entropy regularization is not just a heuristic for exploration; it mathematically transforms the policy gradient into a form of advantage function learning.
  • Equivalence of Architectures: The paper establishes that a regularized actor-critic method is essentially equivalent to a "dueling" action-value architecture.
  • Complementary Updates: The on-policy and off-policy updates are complementary. The on-policy update ensures the agent stays consistent with its current experience, while the off-policy Q-learning update pushes the agent toward global optimality using a wider, historical pool of data.

Results

PGQL demonstrated significant improvements in both data efficiency and overall performance:

  • Superior Performance: On the Atari suite, PGQL achieved the best performance in 34 out of 50+ games, compared to only 7 for A3C and 10 for Q-learning.
  • Data Efficiency: In the Grid World and many Atari games (like Assault and Battle Zone), PGQL learned significantly faster than A3C, reaching higher performance levels with fewer environment steps.
  • Human-Level Threshold: PGQL achieved a median normalized score of 145.6% across all Atari games from random starts, comfortably exceeding the human-level performance threshold (100%). In contrast, A3C (107.3%) and Q-learning (58.9%) trailed behind.

Limitations and Future Work

While highly effective, PGQL has a few limitations:

  • Hyperparameter Sensitivity: PGQL underperformed on a few specific games (like Breakout and Hero). The authors suggest this might be due to the fixed weighting ($\eta$) between the on-policy and off-policy updates, which may require per-game tuning.
  • Local Optima: In cases where PGQL failed, the policy might have overfit to early data or reached a local optimum before the Q-learning updates could provide a meaningful correction.
  • Complex Theory vs. Simple Practice: The theoretical proofs rely on the "tabular" case (where every state is unique). While the algorithm works well in practice with deep neural networks (function approximation), the theoretical guarantees are less rigorous in high-dimensional settings.

Summary updated

Abstract

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer. This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values. This connection allows us to estimate the Q-values from the action preferences of the policy, to which we apply Q-learning updates. We refer to the new technique as ‘PGQL’, for policy gradient and Q-learning. We also establish an equivalency between action-value fitting techniques and actor-critic algorithms, showing that regularized policy gradient techniques can be interpreted as advantage function learning algorithms. We conclude with some numerical examples that demonstrate improved data efficiency and stability of PGQL. In particular, we tested PGQL on the full suite of Atari games and achieved performance exceeding that of both asynchronous advantage actor-critic (A3C) and Q-learning.
BibTeX
@inproceedings{odonoghue2017combining,
  author = {Brendan O'Donoghue and Remi Munos and Koray Kavukcuoglu and Volodymyr Mnih},
  title = {{Combining policy gradient and Q-learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=B1kJ6H9ex},
}
Figure