Error Controlled Actor-Critic Method to Reinforcement Learning

Xingen Gao, Fei Chao, Changle Zhou, Zhen Ge, Chih‐Min Lin, Longzhi Yang, Xiang Chang, Changjing Shang2021

TL;DR

This paper introduces Error Controlled Actor-Critic (ECAC), a novel reinforcement learning algorithm that tackles the problem of overestimation in value functions caused by approximation errors. By deriving an upper bound on these errors and constraining the divergence between successive policies, ECAC effectively reduces approximation errors, leading to improved performance on continuous control tasks.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often use function approximation (like neural networks) to handle complex, large, or continuous state spaces. However, this approximation introduces errors in estimating the value of states or state-action pairs. A significant consequence of these approximation errors is the "overestimation phenomenon," where the agent tends to overestimate the value of actions, leading to suboptimal policies and hindering the learning process. Existing methods like Double Q-learning try to mitigate this directly, but this paper focuses on an indirect approach: controlling the approximation error itself within the actor-critic framework.

Methods

The core of ECAC lies in its novel approach to controlling approximation errors. Instead of directly addressing overestimation, ECAC aims to confine the approximation error in the value function. The authors derive an upper bound on the approximation error of the Q-function approximator. They discover that this upper bound can be minimized by keeping the new policy close to the previous policy during training. This is achieved by introducing a Kullback-Leibler (KL) divergence penalty into the actor's objective function. Essentially, the actor is trained not only to maximize expected rewards but also to ensure that its policy doesn't change too drastically between updates. This indirectly stabilizes the critic's learning and reduces approximation errors. ECAC also incorporates clipped double-Q learning within the critic module to further combat overestimation. The algorithm also features an adaptive mechanism to adjust the coefficients of the KL and entropy terms.

Datasets

The experiments were conducted on continuous control tasks from the OpenAI Gym suite, specifically using environments from the MuJoCo and PyBullet physics engines. These include tasks like Hopper, Walker2d, HalfCheetah, Ant, and Humanoid. No new benchmarks were introduced.

Key Insights and Ideas

  • Indirect Error Control: The most significant insight is that controlling approximation error indirectly by limiting policy changes can be more effective than directly addressing overestimation.
  • KL Divergence as a Stabilizer: Restricting the KL divergence between consecutive policies acts as a powerful mechanism to bound and reduce approximation errors in the value function.
  • Policy Smoothness for Critic Stability: By encouraging smoother policy updates, the critic's learning process becomes more stable, leading to more accurate value estimates.
  • Synergy with Existing Methods: ECAC's approach can be combined with existing methods like clipped double-Q learning, offering a more robust solution.

Key Results

  • Performance Improvement: ECAC consistently outperformed other mainstream model-free RL algorithms (A2C, PPO, TD3, SAC) on most of the tested continuous control tasks.
  • Reduced Approximation Error: Ablation studies showed that ECAC significantly reduced the approximation error in Q-functions compared to a baseline without KL limitation.
  • Lower KL Divergence: The KL divergence between consecutive policies remained lower throughout training when using ECAC, confirming the effectiveness of the proposed constraint.
  • Table 1 Summary: ECAC achieved the highest average return on several tasks, often outperforming or being competitive with TD3 and SAC. For instance, on Ant-v3, ECAC achieved 5420.8 ± 1069.6, significantly higher than other methods.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper mentions adaptive coefficients for KL and entropy terms, the initial setup and tuning of these and other hyperparameters (learning rate, discount rate, etc.) are crucial and might require careful experimentation.
  • Generalizability: While tested on continuous control tasks, the applicability to other domains (e.g., discrete action spaces, different types of environments) is not extensively explored.
  • Future Work: The authors suggest that their indirect operation (KL penalty) can work together with direct strategies for diminishing overestimation, implying potential for hybrid approaches. Further exploration of different mechanisms for confining KL divergence beyond the specific implementation used could also be a future direction.

Summary updated

Abstract

In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably cause overestimation phenomenon and have a negative impact on the convergence of the algorithms. To mitigate the negative effects of approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures confining the approximation error in value function. In this paper, we firstly present an analysis of how the approximation error can hinder the optimization process of actor-critic methods. Then, we *derive an upper boundary of the approximation error of Q function approximator, and found that the error can be lowered by placing restrictions on the KL-divergence between every two consecutive policies during the training phase of the policy.* The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and significantly outperforms other model-free RL algorithms.
BibTeX
@misc{gao2021error,
  author = {Xingen Gao and Fei Chao and Changle Zhou and Zhen Ge and Chih‐Min Lin and Longzhi Yang and Xiang Chang and Changjing Shang},
  title = {{Error Controlled Actor-Critic Method to Reinforcement Learning}},
  year = {2021},
}
Figure