Abstract
The overestimation phenomenon caused by function approximation is a well-known issue in value-based reinforcement learning algorithms such as deep Q-networks and DDPG, which could lead to suboptimal policies. To address this issue, TD3 takes the minimum value between a pair of critics, which introduces underestimation bias. By unifying these two opposites, we propose a novel Weighted Delayed Deep Deterministic Policy Gradient algorithm, which can reduce the estimation error and further improve the performance by weighting a pair of critics. We compare the learning process of value function between DDPG, TD3, and our proposed algorithm, which verifies that our algorithm could indeed eliminate the estimation error of value function. We evaluate our algorithm in the OpenAI Gym continuous control tasks, outperforming the state-of-the-art algorithms on every environment tested.
Introduction
In value-based Reinforcement Learning (RL), agents attempt to learn a "Q-function" that predicts the long-term reward of taking a specific action in a given state. However, these predictions are rarely perfect due to function approximation errors. Historically, algorithms like DDPG suffered from overestimation bias. Because the agent greedily selects actions with the highest predicted value, random noise or approximation errors that inflate an estimate get reinforced, leading to a cycle of over-optimism and suboptimal policies.
To mitigate this, the TD3 algorithm introduced a "Twin Critic" approach: it maintains two independent value estimates and uses the minimum of the two for updates. While this successfully eliminates overestimation, it introduces a new problem: underestimation bias. By being systematically pessimistic, the agent may avoid potentially high-reward paths, slowing learning or settling for mediocre results.
The WD3 Algorithm
We propose Weighted Delayed Deep Deterministic Policy Gradient (WD3) to find a "Goldilocks" middle ground between over- and underestimation. The core innovation is a novel target update rule that balances the optimism of a standard average with the pessimism of the minimum operator.
Instead of strictly taking the minimum of two critics ($Q_1$ and $Q_2$), WD3 calculates a weighted sum: $$y = r + \gamma \left( \beta \min(Q_{1,2}) + \frac{1-\beta}{2} \sum Q_{1,2} \right)$$ Here, $\beta \in [0, 1]$ is a hyperparameter. If $\beta=1$, the algorithm behaves like TD3. By lowering $\beta$, the agent incorporates more of the mean estimate, counteracting chronic underestimation. WD3 also utilizes Delayed Updates, a technique inherited from TD3 where the policy (actor) is updated less frequently than the value functions (critics) to enhance training stability.
Experimental Results
We evaluated WD3 on a suite of continuous control tasks from the PyBullet and OpenAI Gym environments, including Ant-v0, HalfCheetah, Hopper, and Walker2D.
Value Estimation Accuracy
To verify the reduction in bias, we compared estimated Q-values against "true" values calculated via Monte Carlo methods. In tasks like Ant-v0, DDPG's estimates exhibit extreme overestimation, while TD3's estimates stay significantly below the truth. As shown in Figure 1, WD3's estimates track the true value much more closely, demonstrating its ability to eliminate systematic estimation errors.
Performance Benchmarks
WD3 consistently outperformed or matched state-of-the-art algorithms like TD3, DDPG, PPO, and TRPO across almost all tested environments. Figure 2 displays the learning curves for several tasks, showing WD3's superior sample efficiency and final performance. The algorithm also proved robust to the choice of the weighting parameter $\beta$, maintaining high performance across a range of values.
Conclusion
WD3 addresses the fundamental trade-off between overestimation and underestimation in deep RL. By using a convex combination of the minimum and mean of twin critics, it provides more accurate value estimates, leading to superior performance on standard benchmarks. Future work includes exploring dynamic weighting strategies and investigating the entanglement between estimation bias and exploration.
Summary updated