Executive Summary
Adaptive Estimation Q-learning (AEQ) is a novel reinforcement learning framework that dynamically balances over- and underestimation bias in Q-value functions. By combining ensemble-based uncertainty with a "familiarity" metric, AEQ allows agents to be optimistic about new experiences to encourage exploration while remaining conservative and accurate about well-known actions, leading to state-of-the-art performance on continuous control tasks.
Introduction
In model-free deep reinforcement learning (RL), agents rely on Q-values to estimate the long-term reward of taking a specific action in a given state. However, these estimations are notoriously prone to bias. Standard algorithms like DDPG often suffer from overestimation bias because they use a "max" operator over noisy estimates, leading to unstable policies. To fix this, algorithms like TD3 use "clipped double Q-learning," which often results in underestimation bias. While underestimation is generally safer, it can make an agent too "timid," preventing it from exploring high-value regions.
The authors argue that neither extreme is ideal. Instead, an agent should be able to adapt its bias: being slightly optimistic (overestimating) when encountering something new to encourage exploration, and becoming more realistic or conservative (underestimating) as it gains experience to ensure stability.
Methodology
The authors introduce Adaptive Estimation Q-learning (AEQ), which can be integrated into existing actor-critic algorithms like TD3 and SAC. Its core components are:
- Ensemble Q-learning: Instead of one or two Q-networks, AEQ uses an ensemble of $N$ critics (e.g., $N=10$). This allows the agent to calculate both a mean value ($\bar{Q}$) and the Uncertainty ($\hat{\sigma}$), represented by the standard deviation across the ensemble.
- Familiarity ($\mathcal{F}$): This is a simple but effective count-based metric. Every time a specific experience is sampled from the replay buffer, its "count" increases. Familiarity is the ratio of an experience's count to the maximum count seen so far.
- Adaptive Penalty: The Q-target is calculated by taking the mean of the ensemble and subtracting a penalty term:
$$y = r + \gamma [ \bar{Q} - \beta_b \hat{\sigma} - \beta_s \mathcal{F} \hat{\sigma} ]$$
- When an experience is new (low familiarity), the penalty is small, allowing for potential optimism.
- As the experience becomes familiar, the penalty increases, neutralizing overestimation and pushing the estimate toward a more accurate or conservative value.
The authors theoretically prove that there exists a specific familiarity level where the expected estimation bias is approximately zero.
Experimental Results
The researchers evaluated AEQ on the MuJoCo (Multi-Joint dynamics with Contact) suite, a standard benchmark for continuous control in robotics simulation. Specifically, they tested on HalfCheetah-v2, Ant-v2, Walker2d-v2, Hopper-v2, and Swimmer-v2.
Key Findings
- Outperforming the State-of-the-Art: AEQ-TD3 consistently outperformed standard TD3, SAC, and PPO across all MuJoCo environments. For example, in the Walker2d task, AEQ-TD3 achieved a score of ~5236 compared to TD3's ~4266.
- Superior Sample Efficiency: When applied to SAC (AEQ-SAC) and compared against REDQ (a high-performance ensemble method), AEQ achieved higher returns with the same amount of data, particularly on the challenging Ant and Walker2d tasks.
- Bias Stabilization: Experimental plots confirmed that while standard TD3 remains heavily underestimated throughout training, AEQ successfully brings the estimation bias closer to zero as the agent becomes more familiar with the environment.
Discussion and Limitations
AEQ demonstrates that bias can be a tool for balancing exploration and safety. By making bias adaptive to specific state-action pairs, the agent uses optimism for exploration and pessimism for stability. AEQ is simple and general, requiring no complex density models and being easily "pluggable" into off-policy actor-critic algorithms.
However, there are some limitations to consider:
- Computational Overhead: Because AEQ uses an ensemble of 10 critics, it requires more memory and computational power during training.
- Familiarity Metric: The current familiarity term is based on a simple sample counter. More sophisticated density models could make the familiarity metric even more robust.
- Hyperparameter Sensitivity: AEQ introduces two new hyperparameters ($\beta_b$ and $\beta_s$) that control the weight of the uncertainty and familiarity penalties, which may require tuning for new environments.
Summary updated