Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control

Chao Li, Chen Gong, Qiang He, Xinwen Hou

Abstract

Reinforcement learning (RL) often struggles with function approximation error and sample inefficiency. While ensemble methods mitigate bias, they frequently suffer from "ensemble collapse," where sub-policies converge to identical behaviors. We present TEEN (Trajectories-awarE Ensemble exploratioN), an algorithm that explicitly promotes trajectory diversity within an ensemble. By maximizing the discrepancy between state-action visit distributions, TEEN ensures broader exploration. Our evaluation on MuJoCo and DeepMind Control Suite benchmarks shows that TEEN significantly outperforms existing ensemble and off-policy baselines.

Introduction

Deep Reinforcement Learning (DRL) has demonstrated success in complex domains, yet it remains hampered by overestimation bias and high sample complexity. Overestimation bias occurs when function approximation errors lead an agent to overvalue certain actions, resulting in suboptimal policies. Sample inefficiency makes training prohibitively expensive for real-world applications.

Ensemble methods have been proposed to address these issues by averaging multiple models. However, standard ensembles often fail to maintain diversity; sub-policies typically "collapse" into the same behavior, wasting the potential for parallel exploration. This paper addresses the challenge of maintaining diverse exploration throughout the learning process.

Methodology: Trajectory-Aware Exploration

The core innovation of TEEN is Trajectory-aware Exploration. Unlike methods that only encourage diverse actions at specific states, TEEN optimizes for diverse sequences of states and actions (trajectories).

Discrepancy via Mutual Information

TEEN utilizes the Kullback-Leibler (KL) divergence to measure the difference between the state-action visit distributions of sub-policies. To make this computationally tractable, the algorithm employs a neural network discriminator trained to identify which sub-policy generated a given state-action pair. High discriminator accuracy implies that sub-policies are successfully exploring distinct regions of the environment.

Balancing Bias and Variance

To control estimation bias, TEEN integrates two strategies:

  1. Mean Operator: Averaging values across the ensemble to reduce variance.
  2. Min Operator: Applying a minimum operator over a subset of Q-functions to suppress overestimation.

Furthermore, TEEN employs a recurrent optimization schedule, updating one sub-policy at a time over fixed intervals to prevent destructive interference between ensemble members.

Experimental Results

We evaluated TEEN against several baselines, including SUNRISE (a leading ensemble method), SAC, and TD3, across two major benchmark suites: MuJoCo and DeepMind Control Suite (DMControl).

MuJoCo Benchmarks

In the MuJoCo suite, TEEN consistently achieved higher average returns across tasks such as HalfCheetah, Hopper, Walker2d, and Ant.

Average return on MuJoCo benchmarks. TEEN (blue) consistently outperforms baselines like SAC and SUNRISE across various environments.

DeepMind Control Suite (DMControl)

The performance gains were also evident in the DMControl suite, where TEEN demonstrated superior sample efficiency and final performance in environments like cheetah-run and walker-run.

Average return on DeepMind Control Suite benchmarks. TEEN shows significant improvements in complex tasks like walker-run and fish-swim.

Key Findings

  • Performance Gains: TEEN outperformed the SUNRISE ensemble baseline by an average of 41% and surpassed state-of-the-art off-policy algorithms (SAC, TD3) by 7.3%.
  • Exploration Quality: Visualization via t-SNE confirmed that TEEN’s sub-policies cover a significantly larger and more distinct region of the state space compared to baselines.
  • Scalability: TEEN showed robust performance in the high-dimensional Humanoid-v3 task (376-dimensional state space), indicating its effectiveness for complex robotic control.

Discussion and Conclusion

TEEN demonstrates that explicit trajectory diversity is a powerful tool for improving RL exploration. By treating diversity as an optimization objective rather than a byproduct of initialization, TEEN maintains a "team of rivals" that effectively covers the environment.

Limitations

Despite its success, TEEN is sensitive to reward scaling. If extrinsic rewards are disproportionately large, the diversity objective may be overshadowed; conversely, if rewards are too small, the agent may prioritize diversity over task completion. Future work will focus on automated hyperparameter tuning and adaptive normalization to improve robustness across varying reward scales.

Summary updated

BibTeX
@inproceedings{li2023keep,
  author = {Chao Li and Chen Gong and Qiang He and Xinwen Hou},
  title = {{Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=A6JDQDv7Nt},
}
Figure