Generalized Off-Policy Actor-Critic

Shangtong Zhang, Wendelin Boehmer, Shimon Whiteson

This paper introduces a significant theoretical and practical advancement in off-policy reinforcement learning (RL). By rethinking the fundamental objective that off-policy algorithms try to optimize, the authors bridge the gap between how an agent learns from "behavior" data and how it actually performs when deployed.

The authors propose the counterfactual objective, a new framework that unifies existing off-policy objectives and better predicts a policy's real-world performance. From this, they derive the Geoff-PAC algorithm, which marks the first successful application of "emphatic" RL methods to complex, high-dimensional deep RL benchmarks like Mujoco.

The Problem: The Excursion Trap

In off-policy RL, an agent learns a target policy while following a different behavior policy. In "continuing" tasks (tasks without a natural end, like a robot walking indefinitely), most existing algorithms optimize the excursion objective. This measures how the target policy performs during short "excursions" from the states typically visited by the behavior policy.

The problem is that the excursion objective can be misleading. A policy might look highly effective when starting from the behavior policy's favorite states, but fail miserably when running on its own because it drifts into entirely different, dangerous parts of the state space. Ideally, we want to optimize the alternative life objective—how the policy performs under its own state distribution. However, this is notoriously difficult to estimate without actually running the policy, which defeats the purpose of off-policy learning.

Methodology: The Counterfactual Objective

To resolve this, the authors introduce the counterfactual objective, which uses a new parameter ($\hat{\gamma}$) to interpolate between the excursion objective ($\hat{\gamma}=0$) and the alternative life objective ($\hat{\gamma}=1$).

To operationalize this objective, the paper contributes three key advancements:

  1. The GOPPG Theorem: A "Generalized Off-Policy Policy Gradient" theorem that provides the mathematical blueprint for calculating the gradient of this new objective.
  2. The Emphatic Approach: They use an "emphatic" weighting system. This involves keeping track of a "follow-on" trace that emphasizes updates for states that are likely to be visited under the target policy's distribution.
  3. Geoff-PAC (Generalized Off-Policy Actor-Critic): An actor-critic algorithm that uses these gradients. It learns a density ratio to estimate the difference between the behavior and target distributions and uses this to weight its policy updates.

Experimental Results

The authors evaluated their approach on two fronts: a custom tabular environment designed to expose the flaws of the excursion objective, and standard continuous control benchmarks.

Solving the Toy Problem

Through the "Two-circle MDP," the authors demonstrate that standard off-policy algorithms can converge to a sub-optimal policy because they are "blind" to the state visitation frequency of the target policy.

Performance of Geoff-PAC versus ACE on the Two-circle MDP toy problem.

In this environment, standard algorithms (like ACE) converged to the wrong policy. Geoff-PAC, however, successfully found the optimal policy by correctly accounting for state visitation. The counterfactual objective acts as a spectrum: by tuning $\hat{\gamma}$, researchers can choose how much they want to account for the "counterfactual" state distribution of the target policy.

Deep RL Performance

Historically, emphatic algorithms (which use traces to correct for distribution shifts) were considered too high-variance or unstable for deep RL. This paper proves that with the right objective and proper implementation, they can scale to complex tasks.

Learning curves for Geoff-PAC, DDPG, and TD3 across five Mujoco continuous control benchmarks.

On Mujoco tasks (HalfCheetah-v2, Walker2d-v2, Hopper-v2, Reacher-v2, and Swimmer-v2), Geoff-PAC significantly outperformed its predecessors and showed remarkable stability. When compared to popular off-policy algorithms like DDPG, Geoff-PAC avoided the severe performance drops that DDPG often suffered from as training progressed. Because DDPG ignores the discrepancy between behavior and target distributions, it can become unstable; Geoff-PAC's objective keeps it steady.

Limitations and Future Work

While Geoff-PAC represents a major step forward, the authors note a few limitations:

  • Convergence Theory: While the algorithm performs well empirically, a formal convergence analysis for Geoff-PAC with non-linear function approximation (like neural networks) remains an open theoretical challenge.
  • Hyperparameter Sensitivity: The introduction of $\hat{\gamma}$ and emphatic traces adds new hyperparameters that require tuning, which can be more complex than standard actor-critic setups.
  • Sample Efficiency: Geoff-PAC is more "correct" in its objective, but the paper does not focus on beating the sample efficiency of state-of-the-art methods like Soft Actor-Critic (SAC). Future work may look to combine the generalized objective with maximum entropy RL techniques to improve sample efficiency.

Summary updated

Abstract

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.
BibTeX
@inproceedings{zhang2019generalized,
  author = {Shangtong Zhang and Wendelin Boehmer and Shimon Whiteson},
  title = {{Generalized Off-Policy Actor-Critic}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=d45_ui9-E88},
}
Figure