A Closer Look at Deep Policy Gradients

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander MadryICLR 2020 · OpenReview

The Disconnect in Deep RL

Deep Reinforcement Learning (RL) has achieved incredible feats, from mastering Go to controlling complex robotic systems. However, the field is notorious for "brittleness"—algorithms are often hypersensitive to hyperparameter choices, lack consistency, and are difficult to reproduce.

This paper reveals a startling disconnect between the theoretical foundations of deep policy gradient algorithms (like PPO and TRPO) and how they actually behave in practice. The authors tackle a fundamental question: To what degree does current practice in deep RL reflect the principles informing its development? They investigate whether the "primitives" we assume are working—gradient estimation, value prediction, and reward landscapes—actually function as the theory suggests, and find that agents often succeed despite having poor gradient estimates, value networks that fail to model the true value function, and optimization landscapes that are frequently misleading.

Investigating the Primitives

Instead of proposing a new algorithm, the authors perform a fine-grained empirical analysis of two state-of-the-art methods: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO).

To do this, they compare the approximations used during standard training (typically using ~2,000 samples) against "ground truth" values computed using massive amounts of data (up to $10^6$ or $10^7$ samples). Using standard continuous control benchmarks from the MuJoCo physics engine (Humanoid-v2, Walker2d-v2, and Hopper-v2), they measure:

  • Gradient Quality: How well the estimated gradient correlates with the "true" expected gradient.
  • Value Prediction: How accurately the value network fits the actual expected returns.
  • Landscape Analysis: Visualizing the 3D optimization surfaces of both the "surrogate" objective (what the algorithm optimizes) and the "true" reward.

Key Insights: Success Despite Failure

  • The "True" Gradient is a Ghost: Theoretically, we use stochastic gradient ascent to find better policies. However, the authors found that the gradient estimates used in practice often have near-zero or even negative correlation with the true gradient, especially as tasks get harder or training progresses.
  • Value Networks are Good Students, Bad Models: Value networks are used to reduce gradient variance. The authors found that while these networks successfully minimize their training loss (supervised learning), they do not actually fit the true value function. They are often off by 50% or more.
  • The Landscape is a Lie: Policy gradient methods often optimize a "surrogate" objective because the true reward landscape is hard to estimate. The paper shows that late in training, the surrogate objective and the true reward landscape become misaligned. In some cases, taking a step that increases the surrogate objective actually decreases the true reward.
  • The Core Paradox: Perhaps the most surprising insight is that agents still manage to improve and reach high rewards even when their gradients are noisy and their objectives are misaligned. This suggests our current understanding of why these algorithms work is fundamentally incomplete.

Key Results

Gradient Correlation: In complex tasks, the cosine similarity between the estimated gradient and the true gradient is frequently very low.

Evolution of gradient cosine similarity across training iterations. At standard batch sizes (dashed line), correlation drops near zero as training progresses.

Variance Reduction: Using the "true" value function as a baseline reduces gradient variance significantly more than using the learned value network. However, even the "marginal" improvement from the learned network is critical—agents trained without any baseline perform an order of magnitude worse.

Landscape Visualization: In low-sample regimes, the reward landscape is "jagged" and "untamed," making direct optimization nearly impossible without the smoothing effects of massive batch sizes.

Moving Forward

The authors acknowledge that while they have pinpointed where theory and practice diverge, they haven't yet provided a new theory to bridge the gap. Future directions must include developing more robust gradient estimators that don't decay in quality as training progresses, and investigating why agents remain "successful" despite the poor quality of their underlying primitives.

Ultimately, the field needs to move toward a "multi-faceted understanding" rather than just chasing higher scores on leaderboards. We need better evaluation metrics that look under the hood of the optimization process itself.

Summary updated

BibTeX
@inproceedings{ilyas2020closer,
  author = {Andrew Ilyas and Logan Engstrom and Shibani Santurkar and Dimitris Tsipras and Firdaus Janoos and Larry Rudolph and Aleksander Madry},
  title = {{A Closer Look at Deep Policy Gradients}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=4et2HegsJ8v},
}
Figure