Abstract
In temporal-difference reinforcement learning algorithms, variance in value estimation can cause instability and overestimation of the maximal target value. Many algorithms have been proposed to reduce overestimation, including several recent ensemble methods, however none have shown success in sample-efficient learning through addressing estimation variance as the root cause of overestimation. In this paper, we propose MeanQ, a simple ensemble method that estimates target values as ensemble means. Despite its simplicity, MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark. Importantly, we find that an ensemble of size 5 sufficiently reduces estimation variance to obviate the lagging target network, eliminating it as a source of bias and further gaining sample efficiency. We justify intuitively and empirically the design choices in MeanQ, including the necessity of independent experience sampling. On a set of 26 benchmark Atari environments, MeanQ outperforms all tested baselines, including the best available baseline, SUNRISE, at 100K interaction steps in 16/26 environments, and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in 21/26 environments, and by 49% on average, and achieves average human-level performance using 200K (±100K) interaction steps.
Introduction
Training deep reinforcement learning (RL) agents is notoriously difficult due to the "deadly triad": the combination of off-policy learning, function approximation (deep networks), and bootstrapping. This combination often leads to instability and significant overestimation bias. In algorithms like Deep Q-Learning (DQN), the agent uses a max operator to select the best future action. When the underlying value estimates are noisy or have high variance, this operator consistently selects the highest outliers, leading to a "winner's curse" where the agent becomes overconfident in poor actions.
Traditionally, researchers have mitigated this instability using target networks—lagging copies of the main network used to provide stable training targets. However, target networks introduce "lag bias," forcing the agent to learn from outdated information and slowing down the learning process. This paper introduces MeanQ, an ensemble-based approach that addresses estimation variance directly, allowing for the removal of target networks and achieving superior sample efficiency.
The MeanQ Algorithm
MeanQ is built on the premise that reducing the variance of the value estimator is the most effective way to stabilize training and mitigate overestimation. The algorithm utilizes an ensemble of $K$ deep Q-networks (typically $K=5$) and introduces three core architectural changes:
1. Ensemble Mean Targets
Instead of relying on a single network or a lagging target network, MeanQ calculates the target value by taking the average prediction across the entire ensemble:
$$V(s) = \max_a \{ \text{mean}_k Q_{\theta_k}(s, a) \}$$
By averaging the predictions, the algorithm smooths out the noise inherent in individual networks, providing a more stable and accurate signal for the max operator.
2. Independent Experience Sampling
A critical component of MeanQ’s success is independent experience sampling. While many ensemble methods share a single mini-batch of data across all networks to reduce computational load, MeanQ requires each network in the ensemble to sample its own independent mini-batch from the replay buffer. This ensures that the networks remain decorrelated. If the networks were to see the same data in the same order, they would likely converge to the same errors, defeating the variance-reduction purpose of the ensemble.
3. Eliminating the Target Network
Because the ensemble mean provides such a stable estimate, the researchers found that the traditional lagging target network is no longer necessary. MeanQ updates its targets using the current weights of the ensemble. This eliminates lag bias, allowing the agent to incorporate new information immediately and significantly accelerating the learning curve.
Experimental Results
The performance of MeanQ was evaluated on 26 games within the Atari Learning Environment (ALE), a standard benchmark for sample-efficient RL. The results demonstrate that MeanQ is significantly more efficient than previous state-of-the-art methods:
- Superiority at 100K Steps: Compared to SUNRISE, the previous leading baseline, MeanQ achieved higher scores in 16 out of 26 environments, outperforming it by an average of 68%.
- Comparison to Rainbow DQN: At 500,000 interaction steps, MeanQ outperformed the robust Rainbow DQN in 21 out of 26 environments, with a 49% higher average return.
- Human-Level Milestone: MeanQ reached average human-level performance across the test suite in approximately 200,000 steps. For context, many standard deep RL algorithms require millions of steps to reach this level of proficiency.
Discussion and Conclusion
The success of MeanQ suggests that variance, rather than the max operator itself, is the primary villain in deep Q-learning instability. By using a small ensemble of five networks and ensuring diversity through independent sampling, MeanQ tames the "deadly triad" without the need for artificial slowing mechanisms like target networks.
The primary trade-off for this efficiency is computational cost. Training five networks and performing independent sampling increases GPU memory usage and processing time per environment step. However, in many real-world applications where environment interaction is expensive or slow, the massive gain in sample efficiency makes MeanQ a highly attractive alternative to traditional single-network methods. Future research may focus on reducing the computational overhead of such ensembles while maintaining the diversity required for variance reduction.
Summary updated