Adaptive Estimation Q-learning with Uncertainty and Familiarity

Xiaoyu Gong, Shuai Lü, Jiayu Yu, Sheng Zhu, Zongze Li

Executive Summary

Adaptive Estimation Q-learning (AEQ) is a novel reinforcement learning framework that dynamically balances over- and underestimation bias in Q-value functions. By combining ensemble-based uncertainty with a "familiarity" metric, AEQ allows agents to be optimistic about new experiences to encourage exploration while remaining conservative and accurate about well-known actions, leading to state-of-the-art performance on continuous control tasks.

Introduction

In model-free deep reinforcement learning (RL), agents rely on Q-values to estimate the long-term reward of taking a specific action in a given state. However, these estimations are notoriously prone to bias. Standard algorithms like DDPG often suffer from overestimation bias because they use a "max" operator over noisy estimates, leading to unstable policies. To fix this, algorithms like TD3 use "clipped double Q-learning," which often results in underestimation bias. While underestimation is generally safer, it can make an agent too "timid," preventing it from exploring high-value regions.

The authors argue that neither extreme is ideal. Instead, an agent should be able to adapt its bias: being slightly optimistic (overestimating) when encountering something new to encourage exploration, and becoming more realistic or conservative (underestimating) as it gains experience to ensure stability.

Methodology

The authors introduce Adaptive Estimation Q-learning (AEQ), which can be integrated into existing actor-critic algorithms like TD3 and SAC. Its core components are:

  1. Ensemble Q-learning: Instead of one or two Q-networks, AEQ uses an ensemble of $N$ critics (e.g., $N=10$). This allows the agent to calculate both a mean value ($\bar{Q}$) and the Uncertainty ($\hat{\sigma}$), represented by the standard deviation across the ensemble.
  2. Familiarity ($\mathcal{F}$): This is a simple but effective count-based metric. Every time a specific experience is sampled from the replay buffer, its "count" increases. Familiarity is the ratio of an experience's count to the maximum count seen so far.
  3. Adaptive Penalty: The Q-target is calculated by taking the mean of the ensemble and subtracting a penalty term: $$y = r + \gamma [ \bar{Q} - \beta_b \hat{\sigma} - \beta_s \mathcal{F} \hat{\sigma} ]$$
    • When an experience is new (low familiarity), the penalty is small, allowing for potential optimism.
  4. As the experience becomes familiar, the penalty increases, neutralizing overestimation and pushing the estimate toward a more accurate or conservative value.

The authors theoretically prove that there exists a specific familiarity level where the expected estimation bias is approximately zero.

Experimental Results

The researchers evaluated AEQ on the MuJoCo (Multi-Joint dynamics with Contact) suite, a standard benchmark for continuous control in robotics simulation. Specifically, they tested on HalfCheetah-v2, Ant-v2, Walker2d-v2, Hopper-v2, and Swimmer-v2.

Performance of AEQ-TD3 compared to standard TD3, SAC, and PPO across all MuJoCo environments.

Key Findings

  • Outperforming the State-of-the-Art: AEQ-TD3 consistently outperformed standard TD3, SAC, and PPO across all MuJoCo environments. For example, in the Walker2d task, AEQ-TD3 achieved a score of ~5236 compared to TD3's ~4266.
  • Superior Sample Efficiency: When applied to SAC (AEQ-SAC) and compared against REDQ (a high-performance ensemble method), AEQ achieved higher returns with the same amount of data, particularly on the challenging Ant and Walker2d tasks.
Sample efficiency comparison of AEQ-SAC against REDQ and standard SAC on Walker2d and Ant environments.
  • Bias Stabilization: Experimental plots confirmed that while standard TD3 remains heavily underestimated throughout training, AEQ successfully brings the estimation bias closer to zero as the agent becomes more familiar with the environment.

Discussion and Limitations

AEQ demonstrates that bias can be a tool for balancing exploration and safety. By making bias adaptive to specific state-action pairs, the agent uses optimism for exploration and pessimism for stability. AEQ is simple and general, requiring no complex density models and being easily "pluggable" into off-policy actor-critic algorithms.

However, there are some limitations to consider:

  • Computational Overhead: Because AEQ uses an ensemble of 10 critics, it requires more memory and computational power during training.
  • Familiarity Metric: The current familiarity term is based on a simple sample counter. More sophisticated density models could make the familiarity metric even more robust.
  • Hyperparameter Sensitivity: AEQ introduces two new hyperparameters ($\beta_b$ and $\beta_s$) that control the weight of the uncertainty and familiarity penalties, which may require tuning for new environments.

Summary updated

Abstract

One of the key problems in model-free deep reinforcement learning is how to obtain more accurate value estimations. Current most widely-used off-policy algorithms suffer from over- or underestimation bias which may lead to unstable policy. In this paper, we propose a novel method, Adaptive Estimation Q-learning (AEQ), which uses uncertainty and familiarity to control the value estimation naturally and can adaptively change for specific state-action pair. We theoretically prove the property of our familiarity term which can even keep the expected estimation bias approximate to 0, and experimentally demonstrate our dynamic estimation can improve the performance and prevent the bias continuously increasing. We evaluate AEQ on several continuous control tasks, outperforming state-of-the-art performance. Moreover, AEQ is simple to implement and can be applied in any off-policy actor-critic algorithm.
BibTeX
@inproceedings{gong2023adaptive,
  author = {Xiaoyu Gong and Shuai Lü and Jiayu Yu and Sheng Zhu and Zongze Li},
  title = {{Adaptive Estimation Q-learning with Uncertainty and Familiarity}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/417},
}
Figure