Abstract
In the context of learning deterministic policies in continuous domains, we revisit an approach first proposed in Continuous Actor Critic Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic (NFAC). This approach utilizes a policy update distinct from the standard deterministic policy gradient (DPG). We provide a theoretical explanation to motivate this unorthodox update by relating it to the stochastic policy gradient and making explicit its objective function. Furthermore, we introduce Penalized NFAC (PeNFAC), a new trust region algorithm that incorporates these insights. Experimental results across several classic control problems demonstrate that PeNFAC surpasses state-of-the-art algorithms in learning deterministic policies.
Introduction
Reinforcement learning (RL) agents in continuous action spaces, such as those controlling robotic joint torques, often aim to learn deterministic policies. The standard approach is the Deterministic Policy Gradient (DPG), popularized by algorithms like DDPG. However, DPG can be brittle; it requires estimating a complex state-action value function ($Q$-function) and is highly sensitive to the accuracy of the $Q$-function's gradient.
An alternative family of algorithms, including CACLA, takes a simpler route: if an exploratory action results in a positive surprise (a positive Temporal Difference error), the policy is moved toward that action. While CACLA has shown strong empirical performance, it previously lacked a solid theoretical foundation. This paper bridges that gap and introduces PeNFAC to stabilize and enhance this approach.
Theoretical Framework: The Power of the Sign
The core innovation lies in updating policies based only on the sign of the advantage function. We prove that CACLA-style updates are approximations of the Stochastic Policy Gradient (SPG) where the magnitude of the advantage is replaced by a Heaviside (step) function.
This "sign-only" approach offers significant robustness. In continuous spaces, moving "away" from a bad action is often ill-defined, but moving "toward" a known good action is clear. By using a Heaviside function, the algorithm effectively ignores actions with negative advantages, preventing the policy from being misled by noisy or inaccurate critic estimates of sub-optimal actions.
The PeNFAC Algorithm
To scale these insights to deep learning, we propose PeNFAC (Penalized Neural Fitted Actor Critic). PeNFAC introduces a trust-region constraint, similar to Proximal Policy Optimization (PPO), by adding a penalty term to the objective function. This ensures that the policy does not change too drastically in a single update, maintaining stability. Additionally, PeNFAC incorporates modern RL enhancements such as $\lambda$-returns and batch normalization.
Experimental Results
We evaluated PeNFAC against DDPG and a deterministic version of PPO on high-dimensional continuous control environments from OpenAI Roboschool, including Hopper, HalfCheetah, and the complex 17-joint Humanoid task.
As shown in the results for RoboschoolHumanoid-v1, PeNFAC achieves substantially higher rewards than both DDPG and PPO. While the baselines struggle to exceed a reward sum of 1000, PeNFAC scales effectively to the 44-state, 17-action space, reaching rewards near 2500.
Component Analysis and Ablation
To understand the drivers of PeNFAC's success, we conducted extensive ablation studies. These tests revealed that $\lambda$-returns and fitted value iteration are the most critical components for performance.
The ablation results on RoboschoolHalfCheetah-v1 demonstrate that removing $\lambda$-returns or fitted V-iteration leads to a significant drop in performance. Interestingly, while batch normalization and the specific choice of CAC over CACLA provide benefits, they are less critical than the core advantage-estimation mechanisms.
Conclusion
By exploiting the sign of the advantage function, PeNFAC provides a robust and high-performing alternative to standard policy gradient methods. While CACLA-style updates may not always converge to the same local optima as DPG, their resilience to noise and high dimensionality makes them a powerful tool for complex continuous control. Future work will explore off-policy extensions and hybrid updates to further improve sample efficiency and convergence properties.
Summary updated