Overestimation Bias in Value-Based Reinforcement Learning

781 concepts459 papers6 proposals

In value-based reinforcement learning (RL), an agent aims to learn the expected long-term reward of taking an action in a given state, represented by the action-value function $Q(s, a)$. To find the optimal policy, the agent must constantly look ahead and select the action that maximizes this value.

However, this maximization step introduces a severe and systematic flaw: overestimation bias. When an agent estimates values using noisy or imperfect approximations, consistently choosing the maximum estimated value leads to an engineered over-optimism. This bias propagates through time, destabilizing training and causing agents to pursue catastrophic or suboptimal strategies.

The Mathematical Root of Overoptimism

To understand why overestimation occurs, consider a state where all available actions have the exact same true value, $Q(s, a) = V^*(s)$. Because our agent uses a function approximator (like a neural network) or learns from limited data, its estimates $\hat{Q}(s, a)$ will not be perfectly accurate. Instead, they will fluctuate around the true value with some random error or noise, $\epsilon(s, a)$:

$$\hat{Q}(s, a) = Q(s, a) + \epsilon(s, a)$$

If the noise $\epsilon$ has a mean of zero, the estimates are unbiased on average. However, the standard Bellman optimality update requires the agent to calculate the target value using a hard maximization operator: $\max_a \hat{Q}(s, a)$.

Mathematically, the expectation of the maximum of a set of random variables is almost always greater than or equal to the maximum of their expectations:

$$\mathbb{E}\left[\max_a \hat{Q}(s, a)\right] \ge \max_a \mathbb{E}\left[\hat{Q}(s, a)\right]$$

By taking the maximum over noisy estimates, the agent systematically selects the actions that have the most positive noise $\epsilon(s, a)$. As Hado van Hasselt et al. demonstrated in their work on Deep Reinforcement Learning with Double Q-Learning, this overestimation is not merely a product of environmental noise; it is inherently triggered by function approximation errors in neural networks. Any complex model trying to fit a value function will have regions where it overshoots, and the max operator will unerringly find and exploit those overshoots.

The Bootstrapping Feedback Loop

If overestimation occurred only once, it might be manageable. The true danger of overestimation bias lies in temporal difference (TD) bootstrapping.

In TD learning, the value of the current state is updated using the estimated value of the next state:

$$Q(s, a) \leftarrow r + \gamma \max_{a'} Q(s', a')$$

If the estimate for $\max_{a'} Q(s', a')$ is biased upward, that overoptimism is baked directly into the target for the current state-action pair $Q(s, a)$. As the agent interacts with the environment, this error cascades backward through the state trajectory.

This creates a destructive feedback loop. The policy optimizes itself to choose actions that maximize the estimated $Q$-values. The critic network, trying to satisfy the Bellman equation, updates its weights to match these inflated targets. As highlighted by Scott Fujimoto in Addressing Function Approximation Error in Actor-Critic Methods, this continuous exploitation of value function inaccuracies causes overestimation errors to propagate rapidly, steering the policy toward highly unstable or divergent behavior.

The problem is exacerbated in specific training regimes. For instance, Yanqiu Wu et al. showed in Aggressive Q-Learning with Ensembles that increasing the update-to-data ratio (updating the network frequently relative to environment steps) accelerates learning but causes overestimation bias to accumulate exponentially. Similarly, Chengyang Ying et al. identified a "Reuse Bias" in On the Reuse Bias in Off-Policy Reinforcement Learning, proving that optimizing a policy repeatedly on the same replay buffer data systematically inflates performance estimates, an issue that cannot be solved by simply adding more data.

Solution 1: Decoupling Selection from Evaluation

The classic remedy for maximization bias is to break the link between the action selection and the action evaluation. In standard Q-learning, the same network parameters $\theta$ are used to both choose the best action and estimate its value:

$$\text{Target} = r + \gamma Q(s', \arg\max_{a'} Q(s', a'; \theta); \theta)$$

Hado van Hasselt et al. solved this with Double DQN, which utilizes two separate sets of weights (an online network $\theta$ and a target network $\theta'$):

$$\text{Target} = r + \gamma Q(s', \arg\max_{a'} Q(s', a'; \theta); \theta')$$

Here, the online network $\theta$ selects the action it thinks is best, but the target network $\theta'$ evaluates that action's value. Because the approximation errors (noise) of $\theta$ and $\theta'$ are largely independent, a positive error in action selection is unlikely to coincide with a positive error in evaluation. This simple decoupling effectively neutralizes the systematic mathematical overestimation.

Solution 2: Pessimism and Ensemble Voting

In continuous action spaces, decoupling via Double DQN is insufficient because the actor network is explicitly trained to find the continuous peak of the critic network, constantly hunting for overestimation cracks.

To counter this, Scott Fujimoto introduced Clipped Double Q-Learning in the TD3 algorithm. Instead of trusting a single critic, TD3 maintains two independent critics, $Q_1$ and $Q_2$. When computing the TD target, it takes the minimum value between them:

$$\text{Target} = r + \gamma \min_{i=1,2} Q_i(s', a')$$

By favoring underestimation, TD3 prevents overestimation errors from spreading. As Fujimoto noted, an underestimated value target is naturally self-correcting: if the agent underestimates an action, it will simply avoid it, meaning that specific error won't be sampled or propagated further. Conversely, an overestimated action is selected repeatedly, reinforcing the error.

This concept was expanded to larger scales by Xinyue Chen et al. in Randomized Ensembled Double Q-Learning (REDQ). Instead of just two networks, REDQ maintains an ensemble of $N$ Q-functions, randomly samples a subset of $M$ functions during the update step, and takes the minimum across that subset. This randomized pessimism provides a highly tunable knob to control bias, allowing agents to train with massive update-to-data ratios without risking value explosion.

However, as explored by Zhizhou Ren et al. in On the Estimation Bias in Double Q-Learning, excessive underestimation can introduce its own pathologies, creating non-optimal fixed points (value saddle points) that trap agents in suboptimal policies because the updates cease to progress toward the true global optimum.

Solution 3: Soft Maximization and Regression

Another elegant way to bypass overestimation bias is to eliminate the hard max operator entirely. If the agent does not perform a hard maximization over noisy estimates, it cannot exploit the peak noise.

Gerhard Neumann et al. pioneered this direction with Locally-Advantage-Weighted Regression (LAWER) in Fitted Q-iteration by Advantage Weighted Regression. Instead of a hard greedy action selection, LAWER uses a "soft-greedy" approach based on advantage-weighted regression. By computing a weighted average of experiences rather than a strict maximum, the algorithm smooths out the impact of function approximation errors, providing stable value estimates and smoother policy transitions in high-dimensional continuous spaces.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 473 papers
  2. Version 1 · 2026-10-04 · 601 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_overestimation_bias_179bed44,
  title = {overestimation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overestimation-bias/rdr_179bed4479f24b729a32f178283c7462},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_overestimation_bias_179bed44_v2,
  title = {overestimation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overestimation-bias/rdr_179bed4479f24b729a32f178283c7462/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake