Taming Underestimation Bias in Reinforcement Learning

39 concepts36 papers1 proposal

Value-based reinforcement learning (RL) agents have long fought a battle against overestimation bias. Because standard Q-learning uses a maximization operator to select target values, it acts like an optimist looking through rose-colored glasses, consistently overestimating rewards by seizing on positive statistical noise.

To counteract this, the field introduced deep pessimism. Algorithms like Double Q-learning and Clipped Double Q-learning (famously used in TD3 and Soft Actor-Critic) decoupled action selection from evaluation or took the strict minimum of multiple critics. However, this correction introduced a new, equally insidious failure mode: underestimation bias. When value functions systematically underestimate true rewards, agents become paralyzed by excessive caution, leading to a phenomenon known as pessimistic underexploration.

Why the Minimum Operator Overcorrects

To understand why modern RL algorithms underestimate value, consider the mechanics of Clipped Double Q-learning. The algorithm maintains two independent Q-value networks, $Q_1(s,a)$ and $Q_2(s,a)$, and constructs its temporal difference target using their minimum: $\min(Q_1, Q_2)$.

Imagine you ask two friends to estimate the value of an asset. Both friends are generally accurate on average, but their estimates contain random noise. If you always choose the lower of the two estimates, your final choice will systematically sit below the asset's true value.

Mathematically, if $X_1$ and $X_2$ are independent random variables drawn from a distribution with mean $\mu$ and variance $\sigma^2$, the expected value of their minimum is strictly less than the true mean:

$$\mathbb{E}[\min(X_1, X_2)] < \mu$$

As Jiang et al. point out, while this clipped approach successfully caps exploding value estimates, it frequently overcorrects into severe underestimation. Because reinforcement learning relies on bootstrapping—updating a current state's value based on the estimated value of the next state—this small negative bias compounds exponentially over a long horizon, dragging down the entire value landscape.

The Cost of Pessimism: Underexploration

Why is underestimation so damaging? Effective reinforcement learning relies heavily on the principle of Optimism Under Uncertainty. If an agent is unsure about the reward of an unexplored state, it should assume a high value to motivate exploration. Underestimation bias flips this principle on its head.

When value estimates are depressed, the agent encounters several critical failure modes:

  • Pessimistic Underexploration: As formalized by Ciosek et al., when an agent uses conservative lower-bound estimates to update its policy, it avoids uncertain actions entirely. The agent assumes that unfamiliar regions of the state space are low-value, creating a feedback loop where it never visits those regions to correct its misestimates.
  • Value Saddle Points: Ren et al. demonstrate that systematic underestimation can trap agents in sub-optimal "value saddle points." The agent becomes stuck in local optima because the paths leading to global optima are erroneously evaluated as worse than the current status quo.
  • Discrete Action Space Collapse: This issue is particularly severe in discrete action spaces. Zhou et al. show that taking the minimum of multiple Q-networks in discrete settings causes extreme underestimation, completely blinding the agent to high-reward trajectories in complex environments like Atari or MOBA games.
  • Timid Offline Exploitation: In offline RL, where agents learn from fixed historical logs, broad penalties are often applied to handle distribution shifts. Yeom et al. argue that this blanket conservatism causes the agent to become too timid to exploit optimal strategies that are explicitly present in the training data.

Mechanisms for Restoring Balance

To escape the trap of excessive pessimism without falling back into value divergence, researchers have developed several core strategies to balance value estimation.

1. Adaptive Weighting and Order Statistics

Instead of rigidly taking the minimum of two critics, algorithms can dynamically adjust the blend of optimism and pessimism based on the current state of learning.

  • Weighted Double Q-learning: Zhang et al. introduced a tunable weighting parameter to interpolate between the overestimation of standard Q-learning and the underestimation of Double Q-learning. Similarly, He et al. proposed adaptively weighting two critics to stabilize value targets.
  • Order Statistics: Tan et al. generalized this approach by utilizing an ensemble of Q-tables. By selecting specific order statistics (rather than just the absolute minimum), their method dynamically tunes parameters to keep estimation bias near zero throughout training.
  • Uncertainty-Driven Weighting: Döring et al. leverage distributional RL, using the sample variance of return distributions to identify local uncertainty and adaptively weight updates between standard and double Q-learning.

2. Alternative Statistical Operators

The minimum operator is highly sensitive to negative outliers. Replacing it with more robust statistical aggregators prevents severe underestimation.

  • Quasi-Medians: Wei et al. introduced the Quasi-Median Operation over an ensemble of critics. By choosing a middle-ground value rather than the minimum, they drastically reduce underestimation bias while maintaining low estimation variance.
  • Power Means: In planning and tree searches, standard arithmetic averaging can underestimate optimal values early on. Dam et al. utilize a power mean backup operator in Monte-Carlo Tree Search, allowing a single parameter to smoothly balance pessimistic averaging and risky maximization.

3. Decoupling Exploration from Exploitation

A key conceptual breakthrough is realizing that the value function used to update the policy does not have to be the same value function used to explore the environment.

                  ┌──────────────────────────────┐
                  │      Ensemble of Critics     │
                  └──────────────┬───────────────┘
                                 │
                ┌────────────────┴────────────────┐
                ▼                                 ▼
┌──────────────────────────────┐   ┌──────────────────────────────┐
│   Conservative Lower Bound   │   │    Optimistic Upper Bound    │
│      (e.g., Min Operator)    │   │     (e.g., Max / Variance)   │
└──────────────┬───────────────┘   └──────────────┬───────────────┘
               │                                  │
               ▼                                  ▼
┌──────────────────────────────┐   ┌──────────────────────────────┐
│     Stable Policy Updates    │   │   Active Action Exploration  │
└──────────────────────────────┘   └──────────────────────────────┘

Ciosek et al. designed Optimistic Actor-Critic (OAC) to exploit this asymmetry. While OAC retains a conservative lower bound for stable critic updates, it constructs an optimistic upper bound to guide the actor's exploration. This allows the agent to remain safe during policy optimization while aggressively testing uncertain actions that might yield high long-term rewards.

4. Selective and Validated Conservatism

In offline settings, pessimism should be applied surgically rather than globally.

  • Exclusive Penalization: Yeom et al. developed Exclusively Penalized Q-learning (EPQ), which uses a threshold mechanism to penalize only the states prone to distribution shift and estimation errors, leaving well-sampled states unpenalized.
  • Validation Buffers: Ostaszewski et al. introduced Validation Pessimism Learning (VPL), which uses a small, dedicated validation buffer to measure actual approximation errors and dynamically tune the agent's pessimism level to match reality.

The evolution of value estimation highlights that managing bias is not about eliminating error entirely, but about finding a statistical equilibrium. Moving away from rigid minimum operators toward adaptive, decoupled, and statistically robust estimators allows modern RL agents to maintain stability without sacrificing the curiosity needed to learn.

Key Foundational Work

Planning under uncertainty, ensembles of disturbance trees and kernelized discrete action spaces (2009) — contributed 1 concept elements to this direction.
Learning from Logged Implicit Exploration Data (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_underestimation_bias_2b7349c7,
  title = {underestimation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/underestimation-bias/rdr_2b7349c7df7d405cab662046ec3eeae9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_underestimation_bias_2b7349c7_v1,
  title = {underestimation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/underestimation-bias/rdr_2b7349c7df7d405cab662046ec3eeae9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake