Introduction
Phasic Policy Gradient (PPG) is a reinforcement learning framework designed to resolve a long-standing trade-off in actor-critic methods: the tension between feature sharing and objective interference. In traditional on-policy algorithms like Proximal Policy Optimization (PPO), researchers often face a dilemma regarding the neural network backbone.
Using a shared network allows the policy to benefit from rich features learned by the value function. However, the competing objectives often lead to "gradient interference," where updates for one objective degrade the other. Furthermore, shared networks force the same level of "sample reuse" (epochs per data point) for both policy and value updates, even though value functions can typically tolerate much higher sample reuse without overfitting. Conversely, using separate networks avoids interference but loses the critical benefits of shared representations, which are especially important in high-dimensional, vision-based environments.
PPG achieves the "best of both worlds" by decoupling the training processes into distinct phases, allowing the value function to be optimized much more aggressively than the policy while still sharing features.
The PPG Framework
PPG operates in two alternating phases to separate policy and value training:
- The Policy Phase: The agent performs standard reinforcement learning (similar to PPO). Crucially, it uses disjoint networks for the policy and the value function to prevent gradient interference. The policy network includes an "auxiliary" value head that isn't used for control but exists to eventually learn shared features.
- The Auxiliary Phase: This phase uses data collected during the policy phase to perform extra optimization. It distills features from the value function into the policy network by optimizing a joint objective: an auxiliary value loss (to train representations) and a behavioral cloning loss (to ensure the policy doesn't drift too far from its original state).
By separating these updates, PPG can perform many more training epochs on the value function during the auxiliary phase without risking the "policy collapse" that often occurs when over-training a policy on old data.
Key Insights
- Asymmetric Sample Reuse: A core insight of PPG is that the optimal amount of training per sample differs for policies and value functions. Policies are sensitive and overfit quickly, whereas value functions are "data-hungry" and benefit from seeing the same transitions multiple times.
- Interference is a Bottleneck: The authors demonstrate that gradient interference in shared-network architectures significantly limits performance. Simply detaching the value function gradient or using disjoint networks yields immediate improvements.
- Periodic Distillation: Instead of joint optimization at every step, periodic "phases" of distillation are sufficient to maintain high-quality shared features without the constant tug-of-war between gradients.
Key Results
The authors evaluated PPG on the Procgen Benchmark, a suite of 16 procedurally generated environments designed to test generalization and sample efficiency in RL with high-dimensional visual inputs.
PPG demonstrates a clear leap in performance over a highly-tuned PPO baseline:
- Superior Efficiency: PPG achieved significantly better sample efficiency across nearly all 16 Procgen environments.
- Mean Performance: On the aggregate "Mean Normalized Performance" metric, PPG reaches a score of ~0.75 in the same timeframe it takes PPO to reach ~0.55.
The authors also introduced a "Single-Network PPG" variant that uses gradient detaching to achieve similar results with the same computational footprint as standard PPO, proving that the benefits of PPG stem from the algorithmic structure rather than simply having more parameters.
Limitations and Future Work
While highly effective, PPG introduces several new hyperparameters, such as the frequency of auxiliary phases and the number of auxiliary epochs. While default settings worked well across Procgen, tuning these for new domains may require effort. Additionally, the default dual-network version takes roughly twice as long to train in wall-clock time compared to PPO (though the Single-Network version mitigates this).
Future work may explore broadening the auxiliary tasks. While this paper focused solely on using the value function as an auxiliary task, the PPG framework could stably incorporate other objectives, such as self-supervised world models, in its phased approach.
Summary updated