Actor Prioritized Experience Replay (Abstract Reprint)

Baturay Saglam, Furkan B. Mutlu, Dogan Can Çiçek, Suleyman S. KozatAAAI 2024 · DOI

Introduction

Prioritized Experience Replay (PER) has long been a cornerstone of discrete reinforcement learning (RL), notably enhancing the efficiency of algorithms like Deep Q-Networks (DQN). By focusing on transitions with high Temporal-Difference (TD) error—moments where the agent's predictions are most inaccurate—PER allows agents to learn more effectively from "surprising" experiences. However, this success has historically failed to translate to off-policy actor-critic frameworks, such as DDPG or SAC, which are commonly used for continuous control tasks. This paper identifies a fundamental theoretical mismatch: transitions with high TD errors, which PER prioritizes, can cause the actor's policy gradient to diverge, leading to significant training instability.

The Problem: The "Surprise" Trap

In an actor-critic setup, the Critic estimates the value of state-action pairs, while the Actor determines the optimal policy based on those estimates. The authors identify a phenomenon termed the "Surprise Trap." In standard PER, the system prioritizes transitions where the Critic's error is highest. While this is beneficial for the Critic's own learning, it is often detrimental to the Actor.

A high TD error indicates that the Critic is currently unreliable regarding a specific transition. If the Actor is trained predominantly on these high-error points, it is essentially following "bad advice" from an uncertain Critic. The authors provide a mathematical demonstration showing that the approximate policy gradient—derived from an imperfect Q-network—diverges from the true gradient when TD errors are excessively large. This explains the poor empirical performance of PER in continuous action domains.

Methodology: Actor Prioritized Experience Replay (APER)

To address this divergence, the authors introduce Actor Prioritized Experience Replay (APER). This framework moves away from the "one-size-fits-all" prioritization of standard PER and introduces three key methodological shifts:

  1. Theoretical Gradient Analysis: The paper proves that the approximate policy gradient doesn't just become noisy under high TD error; it can actually point in a direction that leads the policy away from the optimal path.
  2. Novel Sampling Framework: Instead of sampling based solely on the highest error for both networks, APER introduces a mechanism to balance how the Actor and Critic consume transitions. This ensures the Actor's updates remain stable even when the Critic is encountering high-variance data.
  3. Scheduled Training: The algorithm introduces a "new branch of improvements" by scheduling the training of the Actor and Critic networks differently. This ensures the Actor is not led astray by a Critic that is still highly uncertain about specific state-action pairs, allowing the Critic to "settle" before the Actor heavily samples those transitions.

Key Results and Insights

The proposed APER framework demonstrates several critical advantages over standard uniform sampling and vanilla PER:

  • State-of-the-Art (SOTA) Performance: APER outperforms existing approaches across standard off-policy benchmarks, achieving higher cumulative rewards and faster convergence.
  • Enhanced Stability: By specifically addressing the gradient divergence issue, the framework provides more consistent training runs, mitigating the erratic policy updates common in continuous control tasks using PER.
  • Data Efficiency: The scheduled training approach allows for more efficient use of stored transitions, a vital metric for off-policy reinforcement learning where data collection can be expensive.

Limitations and Future Work

While APER provides a robust solution to the prioritization mismatch, it introduces additional complexity. Implementing the proposed training schedules often requires more hyperparameter tuning compared to vanilla PER. Furthermore, while the abstract focuses on off-policy actor-critic methods (continuous actions), it remains to be seen how this specific "Actor-Prioritized" logic might be refined for purely discrete domains where standard PER is already dominant. The authors suggest that this work opens a new research direction for refining the mathematical relationship between TD error and policy gradient stability.

Summary updated

BibTeX
@inproceedings{saglam2024actor,
  author = {Baturay Saglam and Furkan B. Mutlu and Dogan Can Çiçek and Suleyman S. Kozat},
  title = {{Actor Prioritized Experience Replay (Abstract Reprint)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30610},
}
Figure