Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains

Matthieu Zimmer, Paul Weng

Abstract

In the context of learning deterministic policies in continuous domains, we revisit an approach first proposed in Continuous Actor Critic Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic (NFAC). This approach is based on a policy update different from that of deterministic policy gradient (DPG). Previous work has observed its excellent performance empirically, but a theoretical justification is lacking. To fill this gap, we provide a theoretical explanation to motivate this unorthodox policy update by relating it to another update and making explicit the objective function of the latter. We furthermore discuss in depth the properties of these updates to get a deeper understanding of the overall approach. In addition, we extend it and propose a new trust region algorithm, Penalized NFAC (PeNFAC). Finally, we experimentally demonstrate in several classic control problems that it surpasses the state-of-the-art algorithms to learn deterministic policies.

Introduction

Learning deterministic policies in continuous domains is a critical requirement for robotics, where predictable and safe behavior is paramount. The current industry standard, Deterministic Policy Gradient (DPG/DDPG), relies on a state-action value critic ($Q$-function). However, $Q$-functions are notoriously difficult to estimate accurately in high-dimensional action spaces. Errors in the critic can lead to gradients that misguide the policy.

An alternative approach, Continuous Actor Critic Learning Automaton (CACLA), utilizes a simpler state-value critic ($V$-function) and updates the policy only when an exploratory action yields a positive advantage. While empirically successful, CACLA lacked a formal theoretical foundation. This paper bridges that gap and introduces Penalized Neural Fitted Actor Critic (PeNFAC), a robust algorithm that leverages the sign of the advantage function within a trust-region framework.

Methods and Theoretical Foundation

The authors provide a theoretical bridge between standard policy gradients and "unorthodox" sign-based updates. They demonstrate that as exploration noise approaches zero, the update direction of these methods aligns with the sign of the DPG update. This "sign-based" approach is inherently more robust: if a critic incorrectly estimates a negative advantage, the method simply skips the update rather than following a potentially harmful gradient.

The PeNFAC Algorithm

PeNFAC builds upon Neural Fitted Actor Critic (NFAC) by incorporating:

  1. Batch Learning and $\lambda$-returns: Balancing bias and variance in reward estimation.
  2. Fitted Value Iteration: For more stable critic training.
  3. Trust-Region Constraint: Similar to PPO, an adaptive penalty prevents the policy from changing too drastically in a single step, which is crucial for stability in high-dimensional tasks.

Experimental Results

The researchers evaluated PeNFAC against DDPG and a deterministic version of PPO on high-dimensional Roboschool benchmarks: Hopper-v1, HalfCheetah-v1, and Humanoid-v1.

Performance on Complex Tasks

In the most challenging environment, Humanoid-v1, PeNFAC demonstrated a significant performance advantage. While DDPG often failed to learn a walking gait entirely, PeNFAC converged reliably to high-reward behaviors.

PeNFAC significantly outperforms DDPG and PPO on the complex Humanoid-v1 task.

Ablation Studies

Ablation studies revealed that the "sign-only" update is vital; the algorithm failed to learn when using standard policy gradients. Furthermore, components like fitted value iteration were found to be critical for maintaining performance.

Ablation study showing that removing fitted V iteration leads to a substantial drop in performance on HalfCheetah-v1.

Discussion and Conclusion

PeNFAC offers a robust alternative to DPG-based methods by exploiting the sign of the advantage function. While it shows superior performance in complex locomotion tasks, potential limitations include scaling to extremely high-dimensional action spaces and the lack of global convergence guarantees with non-linear function approximators. Future work may explore off-policy extensions and combining sign-based robustness with the theoretical convergence properties of DPG.

Summary updated

Abstract

In the context of learning deterministic policies in continuous domains, we revisit an approach, which was first proposed in Continuous Actor Critic Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic (NFAC). This approach is based on a policy update different from that of deterministic policy gradient (DPG). Previous work has observed its excellent performance empirically, but a theoretical justification is lacking. To fill this gap, we provide a theoretical explanation to motivate this unorthodox policy update by relating it to another update and making explicit the objective function of the latter. We furthermore discuss in depth the properties of these updates to get a deeper understanding of the overall approach. In addition, we extend it and propose a new trust region algorithm, Penalized NFAC (PeNFAC). Finally, we experimentally demonstrate in several classic control problems that it surpasses the state-of-the-art algorithms to learn deterministic policies.
BibTeX
@inproceedings{zimmer2019exploiting,
  author = {Matthieu Zimmer and Paul Weng},
  title = {{Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=lv5C3Gk2oh},
}
Figure