This paper introduces a significant theoretical and practical advancement in off-policy reinforcement learning (RL). By rethinking the fundamental objective that off-policy algorithms try to optimize, the authors bridge the gap between how an agent learns from "behavior" data and how it actually performs when deployed.
The authors propose the counterfactual objective, a new framework that unifies existing off-policy objectives and better predicts a policy's real-world performance. From this, they derive the Geoff-PAC algorithm, which marks the first successful application of "emphatic" RL methods to complex, high-dimensional deep RL benchmarks like Mujoco.
The Problem: The Excursion Trap
In off-policy RL, an agent learns a target policy while following a different behavior policy. In "continuing" tasks (tasks without a natural end, like a robot walking indefinitely), most existing algorithms optimize the excursion objective. This measures how the target policy performs during short "excursions" from the states typically visited by the behavior policy.
The problem is that the excursion objective can be misleading. A policy might look highly effective when starting from the behavior policy's favorite states, but fail miserably when running on its own because it drifts into entirely different, dangerous parts of the state space. Ideally, we want to optimize the alternative life objective—how the policy performs under its own state distribution. However, this is notoriously difficult to estimate without actually running the policy, which defeats the purpose of off-policy learning.
Methodology: The Counterfactual Objective
To resolve this, the authors introduce the counterfactual objective, which uses a new parameter ($\hat{\gamma}$) to interpolate between the excursion objective ($\hat{\gamma}=0$) and the alternative life objective ($\hat{\gamma}=1$).
To operationalize this objective, the paper contributes three key advancements:
- The GOPPG Theorem: A "Generalized Off-Policy Policy Gradient" theorem that provides the mathematical blueprint for calculating the gradient of this new objective.
- The Emphatic Approach: They use an "emphatic" weighting system. This involves keeping track of a "follow-on" trace that emphasizes updates for states that are likely to be visited under the target policy's distribution.
- Geoff-PAC (Generalized Off-Policy Actor-Critic): An actor-critic algorithm that uses these gradients. It learns a density ratio to estimate the difference between the behavior and target distributions and uses this to weight its policy updates.
Experimental Results
The authors evaluated their approach on two fronts: a custom tabular environment designed to expose the flaws of the excursion objective, and standard continuous control benchmarks.
Solving the Toy Problem
Through the "Two-circle MDP," the authors demonstrate that standard off-policy algorithms can converge to a sub-optimal policy because they are "blind" to the state visitation frequency of the target policy.
In this environment, standard algorithms (like ACE) converged to the wrong policy. Geoff-PAC, however, successfully found the optimal policy by correctly accounting for state visitation. The counterfactual objective acts as a spectrum: by tuning $\hat{\gamma}$, researchers can choose how much they want to account for the "counterfactual" state distribution of the target policy.
Deep RL Performance
Historically, emphatic algorithms (which use traces to correct for distribution shifts) were considered too high-variance or unstable for deep RL. This paper proves that with the right objective and proper implementation, they can scale to complex tasks.
On Mujoco tasks (HalfCheetah-v2, Walker2d-v2, Hopper-v2, Reacher-v2, and Swimmer-v2), Geoff-PAC significantly outperformed its predecessors and showed remarkable stability. When compared to popular off-policy algorithms like DDPG, Geoff-PAC avoided the severe performance drops that DDPG often suffered from as training progressed. Because DDPG ignores the discrepancy between behavior and target distributions, it can become unstable; Geoff-PAC's objective keeps it steady.
Limitations and Future Work
While Geoff-PAC represents a major step forward, the authors note a few limitations:
- Convergence Theory: While the algorithm performs well empirically, a formal convergence analysis for Geoff-PAC with non-linear function approximation (like neural networks) remains an open theoretical challenge.
- Hyperparameter Sensitivity: The introduction of $\hat{\gamma}$ and emphatic traces adds new hyperparameters that require tuning, which can be more complex than standard actor-critic setups.
- Sample Efficiency: Geoff-PAC is more "correct" in its objective, but the paper does not focus on beating the sample efficiency of state-of-the-art methods like Soft Actor-Critic (SAC). Future work may look to combine the generalized objective with maximum entropy RL techniques to improve sample efficiency.
Summary updated