Abstract
We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.
Introduction
In deep reinforcement learning, actor-critic methods are widely used, relying on a policy (the actor) to select actions and a value function (the critic) to estimate expected rewards. A persistent challenge in designing these agents is the "shared representation dilemma." Practitioners must choose between shared networks, which allow for beneficial feature sharing but suffer from gradient interference, and separate networks, which eliminate interference but prevent the actor from leveraging features learned by the critic.
Phasic Policy Gradient (PPG) addresses this trade-off by decoupling policy and value function training into two alternating phases. This framework allows for high-quality feature sharing without the rigid constraints on sample reuse that typically hamper traditional on-policy methods like Proximal Policy Optimization (PPO).
Methods
PPG introduces a training cycle consisting of two alternating phases:
- The Policy Phase: The agent interacts with the environment and updates the policy and value function using disjoint networks (or detached gradients) to prevent interference. This phase is similar to standard PPO training.
- The Auxiliary Phase: This phase facilitates feature distillation. The agent performs an auxiliary optimization using data collected during the policy phase to train an auxiliary value head attached to the policy network's backbone. This forces the policy network to learn the features discovered by the value function. To maintain behavioral stability, a KL-divergence penalty is used to keep the policy close to its state at the end of the policy phase.
By splitting training this way, PPG can optimize the value function much more aggressively than the policy, which is typically more sensitive to over-optimization. This allows for asymmetric sample reuse, where the value function benefits from multiple passes over the same data while the policy remains stable.
Experimental Setup
The authors evaluated PPG using the Procgen Benchmark, a suite of 16 procedurally generated environments. Unlike standard benchmarks, Procgen is designed to test generalization; because levels change every time, agents must learn robust visual features rather than memorizing action sequences. This makes it a rigorous test for the feature-sharing capabilities of PPG.
Key Insights
- Asymmetric Sample Reuse: PPG is the first on-policy method to naturally allow the value function and policy to be trained with different levels of sample reuse, matching their respective "appetites" for data.
- Interference Mitigation: The framework demonstrates that gradient interference in shared-network PPO is a significant performance bottleneck, particularly in complex visual environments.
- Effective Feature Distillation: Periodic distillation of the critic's knowledge into the actor's backbone is sufficient to gain the benefits of shared representations without the instability of joint optimization.
Key Results
PPG significantly outperforms a highly-tuned PPO baseline across the Procgen suite, achieving higher scores with the same amount of environment interaction. The framework also proves robust to hyperparameters, with default values performing well across all 16 diverse environments.
Additionally, PPG offers implementation flexibility. It can be deployed with a single shared network by detaching the value function's gradient during the policy phase, making it a drop-in improvement for existing actor-critic architectures.
Limitations and Future Work
While PPG offers clear performance benefits, it roughly doubles the memory footprint during training due to the use of separate networks. Furthermore, the frequency of the auxiliary phase is a critical hyperparameter; performing it too frequently can reintroduce interference. Future work may explore distilling other auxiliary tasks, such as world models or curiosity-based objectives, into the policy using this phased approach.
Summary updated