In value-based reinforcement learning, agents rely on temporal difference methods to estimate the expected cumulative reward of taking an action in a given state. However, these estimates are notoriously prone to systematic errors.
Standard Q-learning suffers from overestimation bias because its target calculation uses a maximization operator ($\max_a Q(s,a)$). Since neural network function approximators are inherently noisy, taking the maximum over noisy estimates systematically pushes the target value higher than the true value. Conversely, architectures like Double Q-learning or Clipped Double Q-learning (used in TD3) attempt to fix this by decoupling action selection from evaluation or taking the minimum over two critics. This often overcorrects, introducing a severe underestimation bias that makes the agent overly pessimistic and slows down learning.
Rather than accepting either chronic optimism or chronic pessimism, a prominent line of research treats estimation bias as a continuous, controllable spectrum. By introducing tunable mechanisms, practitioners can dial in the exact level of bias needed to stabilize training and maximize sample efficiency.
The Ensemble Dial: Controlling Bias via Model Aggregation
The most direct way to build a bias control knob is by maintaining an ensemble of independent value estimators. By changing how many models are in the ensemble—or how their predictions are aggregated—we can smoothly shift the target value from an overestimate to an underestimate.
[Pure Maximization] <------------------ [Goldilocks Zone] ------------------> [Pure Minimization]
Positive Bias Near-Zero Bias Negative Bias
(Standard Q-Learning) (Tunable Ensembles) (Fixed Double Q-Learning)
In Maxmin Q-learning: Controlling the Estimation Bias of Q-learning, Lan et al. demonstrated that maintaining an ensemble of $N$ independent estimators allows for systematic adjustment of this bias. By taking the minimum value across all $N$ estimators, increasing the ensemble size shifts the bias from positive to negative. This flexibility allows the algorithm to find a "Goldilocks zone" that avoids both the runaway optimism of single-estimator systems and the excessive pessimism of standard double-estimator setups.
However, scaling up the number of estimators can become computationally prohibitive, and taking a strict minimum over a large ensemble can cause catastrophic underestimation. To decouple the total ensemble size from the degree of pessimism, Chen et al. introduced Randomized Ensembled Double Q-Learning (REDQ) in Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. Instead of minimizing over the entire ensemble, the authors select a small, randomized subset of $M$ critics from a larger pool of $N$ models to compute the update target. This architectural choice provides a precise way to manage the bias-variance trade-off, enabling high-frequency updates without incurring debilitating underestimation.
Taking a similar approach to enable aggressive training frequencies, Wu et al. proposed Aggressive Q-Learning with Ensembles (AQE) in Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance. AQE calculates target values by taking the average of the $K$ lowest estimates from an ensemble of $N$ critics. By decreasing $K$, the agent injects a controlled dose of pessimism to counteract the cumulative overestimation typically induced by high update frequencies.
Rather than relying on lower bounds or minimums, Wei et al. explored a central-tendency approach in Controlling Underestimation Bias in Reinforcement Learning via Quasi-median Operation. The authors utilize a "quasi-median" operation over an ensemble of critics, selecting a middle-ground value that naturally balances over-optimistic overestimation and overly cautious underestimation, providing a more stable learning signal than extreme order statistics.
Alternative Knobs: Beyond Network Ensembles
Spawning and training dozens of neural networks is computationally expensive. To circumvent this, several researchers have proposed alternative "knobs" that tune estimation bias without requiring large ensembles of distinct networks.
Distributional Truncation
In Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics, Kuznetsov et al. shifted the bias control mechanism from the network architecture to the return distribution itself. Their method, Truncated Quantile Critics (TQC), drops a specific number of top "atoms" (quantiles) from a predicted return distribution. Varying the number of removed atoms provides a continuous, fine-grained hyperparameter for managing bias that operates independently of the number of networks in the ensemble.
Action Candidate Selection
Instead of modifying the value functions, Jiang et al. focused on the action selection policy in Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Tasks. Their algorithm, AC-CDQ, restricts the target maximization operator to a subset of "elite" action candidates. The size of this candidate set serves as the tunable parameter: increasing the set size shifts the estimator toward standard maximization (optimism), while decreasing it moves the behavior toward conservative double-estimation.
Mathematical Operators and Regularization
In Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning, Behnamnia et al. introduced a Log-Sum-Exponential (LSE) operator to navigate the bias-variance trade-off. By adjusting a continuous scaling parameter within the exponential operator, the estimator can be calibrated to ignore heavy-tailed outliers and stabilize learning.
Similarly, Schmitt-Förster et al. leveraged distributionally robust optimization in Regularized Q-learning through Robust Averaging. By adjusting the size of an uncertainty set alongside a small number of estimators, practitioners can smoothly tune estimation bias along a spectrum to match the noise characteristics of specific environments.
Why It Works: Shifting the Integration Window
To understand why these methods work, it helps to visualize the errors in our value functions as a probability distribution centered around the true value.
When we compute a target value using a standard $\max$ operator, we are effectively sampling from the extreme right tail of the error distribution, causing positive bias. When we take a strict $\min$ over two independent estimators, we sample from the left tail, causing negative bias.
Negative Bias Positive Bias
(Underestimation) (Overestimation)
| |
v v
[ Left Tail | Quasi-Median | Truncated Mean | Right Tail ]
^ ^
|___________________________________|
Tunable Selection Window
Tunable estimation bias works because it allows us to control the "integration window" across this error distribution:
- Averaging the $K$-lowest estimates (Wu et al.) shifts the window slightly to the left of the mean, providing a controlled amount of pessimism.
- Dropping the top $d$ quantiles (Kuznetsov et al.) clips the right tail of the distribution, preventing positive outliers from inflating the value targets.
- Varying the ensemble size acts as a weighted split between action selection and evaluation data. As shown theoretically by Peer et al. in Ensemble Bootstrapping for Q-Learning, increasing the number of models used for averaging reduces the pessimistic bias inherent in decoupled estimation, minimizing the overall Mean Squared Error (MSE) of the updates.
Dynamic and Adaptive Control
While treating bias as a fixed hyperparameter is effective, the optimal amount of bias often changes over the course of training or when transitioning between different learning phases.
In Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback, Wang et al. introduced Adaptive Ensemble Q-learning (AdaEQ). Instead of forcing practitioners to manually tune a fixed ensemble size, AdaEQ uses real-time error feedback to dynamically scale the number of active models up or down. This automated adjustment keeps the net estimation bias near zero throughout the entire learning process.
This adaptive control becomes especially critical in complex training pipelines, such as offline-to-online (O2O) reinforcement learning. As identified by Guo et al. in Tackling Heavy-Tailed Q-Value Bias in Offline-to-Online Reinforcement Learning with Laplace-Robust Modeling, the shift from static offline datasets to live online environments causes the estimation bias to follow an unstable, heavy-tailed distribution. The authors demonstrated that combining heavy-tailed Laplace noise modeling with conservative ensemble-based estimates successfully re-centers the bias distribution mean toward zero, preventing the training instabilities that commonly plague uncalibrated agents.
Key Foundational Work
Ensemble Bootstrapping for Q-Learning (2021) — contributed 4 concept elements to this direction.
Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance (2021) — contributed 3 concept elements to this direction.
Maxmin Q-learning: Controlling the Estimation Bias of Q-learning (2020) — contributed 3 concept elements to this direction.