Generalized Off-Policy Actor-Critic

Shangtong Zhang, Wendelin Boehmer, Shimon Whiteson

Abstract

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.

Introduction: The Problem of Off-Policy Objectives

In off-policy reinforcement learning (RL), an agent learns a "target policy" while following a different "behavior policy." For continuing tasks—those without a natural end—most algorithms optimize the excursion objective. This objective measures performance based on the state distribution visited by the behavior policy.

However, this objective can be deceptive. When the target policy is eventually deployed, it will visit a different distribution of states, a scenario described by the alternative life objective. If the behavior and target policies diverge significantly, optimizing for the excursion can lead to a policy that performs well during training but fails in deployment. Directly optimizing the alternative life objective is difficult because the agent lacks samples from the target policy's future state distribution.

The Counterfactual Objective and Geoff-PAC

To bridge this gap, we introduce the counterfactual objective, which provides a middle ground between the excursion and alternative life objectives.

The Counterfactual Knob ($\hat{\gamma}$)

We introduce a parameter $\hat{\gamma} \in [0, 1]$ to control the degree of "counterfactualness." * When $\hat{\gamma}=0$, the objective reduces to the standard excursion objective. * When $\hat{\gamma}=1$, it recovers the ideal alternative life objective.

Generalized Off-Policy Actor-Critic (Geoff-PAC)

We prove the Generalized Off-Policy Policy Gradient Theorem, establishing the mathematical foundation for calculating gradients of this new objective. To implement this, we employ an emphatic approach. Emphatic algorithms use a special weighting mechanism—the "emphasis"—to ensure gradients remain unbiased during off-policy learning. While historically limited to toy problems, Geoff-PAC scales to deep neural networks by estimating density ratios and value functions.

Figure 2: (a) The Two-circle MDP environment. (b) Learning curves for the target policy $\pi(A \to B)$, comparing Geoff-PAC and ACE. (c) Sensitivity analysis of the counterfactual objective parameters.

Empirical Evaluation

We evaluated Geoff-PAC on two main fronts:

  1. Two-circle MDP: A diagnostic environment where the excursion objective fails. Standard algorithms get trapped in a high-reward "inner circle" because the behavior policy keeps them there. Geoff-PAC successfully identifies that the "outer circle" is superior for long-term deployment.
  2. MuJoCo Robot Simulations: Standard benchmarks (HalfCheetah, Walker2d, Hopper, Reacher, Swimmer) modified into continuing tasks.

Key Results

Geoff-PAC represents a milestone as the first emphatic algorithm to achieve empirical success on deep RL benchmarks. It significantly outperformed standard Off-Policy Actor-Critic (Off-PAC) and Actor-Critic with Emphatic weightings (ACE) in several MuJoCo tasks.

Figure 3: Performance comparison on the Swimmer-v2 environment. Geoff-PAC (blue) demonstrates superior performance compared to baselines.

Furthermore, Geoff-PAC showed remarkable robustness to "off-policyness." When trained with a completely random behavior policy, it maintained stability where popular algorithms like DDPG and TD3 often suffered performance degradation.

Conclusion and Future Work

While Geoff-PAC demonstrates the potential of emphatic methods in deep RL, challenges remain. A full convergence analysis for non-linear function approximation is still needed. Additionally, the algorithm's complexity—requiring estimation of value functions, density ratios, and emphasis—introduces computational overhead and additional hyperparameters. Future research may explore hybrid methods that combine on-policy and off-policy samples to further enhance stability.

Summary updated

Abstract

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.
BibTeX
@inproceedings{zhang2019generalized,
  author = {Shangtong Zhang and Wendelin Boehmer and Shimon Whiteson},
  title = {{Generalized Off-Policy Actor-Critic}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=d45_ui9-E88},
}
Figure