Introduction
In deep reinforcement learning (RL), variance in value estimation frequently leads to severe overestimation bias and training instability. To combat this, researchers introduce MeanQ, a straightforward ensemble method that uses the mean of multiple Q-networks to estimate target values. By drastically reducing variance, MeanQ stabilizes training so effectively that it can eliminate the need for lagging target networks entirely, achieving state-of-the-art sample efficiency on the Atari benchmark.
The Problem: Variance and the "Winner's Curse"
In value-based RL algorithms like Q-learning and DQN, agents learn by iteratively updating their value estimates using bootstrapped targets (the current reward plus the estimated future value). Because these estimates are noisy and uncertain—especially early in training—taking the max over actions introduces a positive bias known as the "winner's curse" or Jensen's gap.
Essentially, variance in your estimates causes the agent to systematically overestimate how good certain actions are, leading to severe training instability. For the past decade, the standard solution has been to use a target network—a delayed, lagging copy of the main network used to calculate targets. While target networks stabilize training, they introduce a lagging bias that slows down learning and reduces sample efficiency. The authors of MeanQ ask: Can we fix the root cause (variance) directly, rather than slapping a band-aid (target networks) on the symptoms?
The MeanQ Approach
MeanQ is a surprisingly simple ensemble-based RL algorithm. Instead of a single Q-network, MeanQ maintains an ensemble of $K$ independent Q-networks (typically $K=5$).
When calculating the target value for a Bellman update, MeanQ takes the mean of the ensemble's predictions for each action, and then takes the max over those averaged values. To make this work effectively, the authors highlight three critical design choices:
- Shared Exploration Policy: The agent explores the environment using an $\epsilon$-greedy policy based on the ensemble mean, ensuring high-quality data collection.
- Shared Replay Buffer: All networks store their experiences in one common pool, maximizing the diversity of available data.
- Independent Replay Sampling: This is the secret to the algorithm's success. To keep the ensemble members decorrelated—which is mathematically required for an average to actually reduce variance—each network samples its own independent mini-batches from the shared replay buffer and maintains its own prioritized experience replay weights.
Because MeanQ naturally suppresses variance, the authors also experiment with dropping the traditional target network entirely.
Key Insights
- Variance = Bias: The paper empirically demonstrates a strong positive correlation between estimation variance and the Jensen gap (overestimation bias). By squashing variance through ensemble averaging, MeanQ directly mitigates overestimation.
- Goodbye Target Networks: Because MeanQ is inherently stable, the lagging target network can be safely discarded. Removing this lag allows the network to learn from up-to-date estimates, significantly accelerating early-stage learning.
- Decorrelation is Crucial: Previous ensemble methods (like Ensemble-DQN) often shared mini-batches across the ensemble to save compute. MeanQ shows that this correlates the networks too much, destroying the variance-reduction benefits. Independent sampling is vital.
- Not Just a "Bigger Brain": Ablation studies prove that MeanQ's success isn't just because 5 networks have 5 times the parameters, or because it does 5 times the gradient updates. A single massive network or a network updated 5x more frequently actually performs worse. The magic specifically lies in ensemble variance reduction.
Key Results
The method was evaluated on the Atari Learning Environment (ALE), focusing on the sample-efficient regime (100K and 500K environment interaction steps).
MeanQ achieves remarkable sample efficiency, outperforming strong baselines:
- At 100K steps: MeanQ outperforms the previous state-of-the-art ensemble method (SUNRISE) in 16 out of 26 Atari games, achieving a 68% higher average normalized return.
- At 500K steps: MeanQ outperforms the highly optimized Rainbow DQN in 21 out of 26 games, with a 49% higher average normalized return.
- Human-Level Performance: Averaged across the 26 games, MeanQ achieves human-level performance using only 200K ($\pm 100K$) interaction steps.
Limitations and Future Work
While highly effective, MeanQ does come with a computational cost. Because each of the $K$ networks samples independently, computing the target values requires evaluating all $K$ members on every state present in any member's mini-batch. This results in $O(K^2)$ target evaluations. While perfectly manageable for $K=5$, it scales poorly for massive ensembles.
Additionally, there are theoretical mysteries left to solve. The authors note that MeanQ's benefits seem to rely on the use of nonlinear function approximation (deep neural networks). In simple linear or tabular RL, a simple mean doesn't provide the same benefits. Ultimately, MeanQ serves as a strong, simple baseline for future research into the complex interplay between variance, bias, exploration, and the "deadly triad" of off-policy reinforcement learning.
Summary updated