Lean Evolutionary Reinforcement Learning by Multitasking with Importance Sampling

Nick Zhang, Abhishek Gupta, Zefeng Chen, Yew-Soon Ong

TL;DR

This paper introduces NuEMT, a novel neuroevolutionary multitasking algorithm that significantly improves the data efficiency of evolution strategies (ES) for reinforcement learning (RL). By leveraging shorter "auxiliary" tasks to transfer learned skills to a main, longer task using importance sampling, NuEMT drastically reduces the amount of environment interaction data needed for training.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by interacting with an environment and maximizing rewards. Evolution Strategies (ES) are a promising, derivative-free approach for training deep neural network policies in RL, especially for continuous control tasks. However, ES suffers from high "sample complexity," meaning it requires a vast amount of agent-environment interactions to learn effectively, particularly for tasks with long horizons (e.g., complex simulations or long-duration tasks). This makes training computationally expensive and time-consuming. Existing methods like "progressive episode lengths" (PEL) attempt to mitigate this by training on tasks with increasing episode lengths sequentially, but this can lead to negative transfer if skills learned on shorter tasks don't generalize well to longer ones.

Methods

The core innovation is the NuEMT (Neuroevolutionary Multitasking) algorithm. It builds upon the OpenAI-ES framework, which uses a population of policies and adds Gaussian noise to explore the policy space. NuEMT extends this by:

  • Multitask Formulation: Instead of training a single task, NuEMT trains a target task alongside multiple "auxiliary" tasks concurrently. These auxiliary tasks are derived from the main task but have shorter episode lengths.
  • Importance Sampling for Skill Transfer: The key novelty is using statistical importance sampling to transfer skills from auxiliary tasks to the main task. This allows the algorithm to learn from policies evolved on shorter episodes and use that knowledge to guide the search for the optimal policy on the longer, main task. This mechanism effectively "transfers" learned skills without the risk of negative transfer inherent in sequential approaches.
  • Adaptive Resource Allocation: NuEMT dynamically allocates computational resources (population size) to auxiliary tasks based on their "usefulness" to the main task. Tasks that contribute more to the main task's learning receive more resources, while less useful tasks are de-emphasized, minimizing wasted computation.
  • Probabilistic Mixture Model: The algorithm uses a mixture model to represent the combined probability distribution of candidate solutions from all tasks, allowing for controlled inter-task information exchange.

Datasets

The paper evaluates NuEMT on standard continuous control tasks from the OpenAI Gym suite, specifically:

  • MuJoCo tasks: HalfCheetah-v2, Swimmer-v2, Hopper-v2, Ant-v2, Walker2d-v2, Humanoid-v2.
  • Box2D simulators: BipedalWalker-v3 with varying leg lengths.

No new benchmarks were introduced; instead, the focus was on demonstrating efficiency on existing, well-established RL environments.

Key Insights and Ideas

  • Multitasking as a Data Efficiency Booster: The central idea is that learning on simpler, shorter tasks can provide valuable "building blocks" of knowledge that accelerate learning on more complex, longer tasks. This is analogous to how humans learn by mastering simpler skills before tackling more advanced ones.
  • Importance Sampling for Controlled Transfer: Using importance sampling is a clever way to bridge the gap between different task distributions (due to varying episode lengths) and enable effective skill transfer without explicit model rebuilding, as seen in some prior EMT approaches.
  • Adaptive Resource Allocation is Crucial: Simply having auxiliary tasks isn't enough; dynamically assigning computational resources based on their actual contribution to the main task is key to maximizing efficiency and avoiding wasted effort on irrelevant skills.
  • NuEMT Outperforms Sequential Transfer: The concurrent nature of multitasking in NuEMT appears to be more robust than sequential transfer methods like PEL, especially in scenarios where skills might not generalize perfectly across episode lengths.

Key Results

  • Significant Data Efficiency Gains: NuEMT consistently outperforms the baseline OpenAI-ES and PEL methods in terms of sample efficiency across most tested MuJoCo and Box2D tasks. For example, in the HalfCheetah-v2 task, NuEMT achieved better performance with fewer timesteps compared to OpenAI-ES.
  • Faster Convergence: NuEMT demonstrates accelerated convergence, reaching higher cumulative rewards earlier in the training process, particularly on the main task.
  • Robustness: NuEMT shows strong performance across various tasks, including those with challenging dynamics like the Humanoid-v2 (where survival bonus was removed) and BipedalWalker-v3 with different leg lengths.
  • Mixture Coefficient Learning Effectiveness: Experiments showed that learning and updating mixture coefficients (as implemented in NuEMT) leads to significantly better performance compared to using fixed mixture coefficients.

Limitations and Future Work

  • Hyperparameter Sensitivity: While not explicitly detailed as a limitation, the paper mentions that the performance can be affected by the number of auxiliary tasks and the fixed population size. The optimal number of auxiliary tasks might be environment-dependent.
  • Generalization Beyond RL: The authors suggest extending the general idea of leveraging simpler tasks to improve learning on complex problems beyond RL, such as in neural architecture search or other machine learning sub-fields.
  • Hardware Efficiency: While NuEMT is computationally cheaper per iteration than conventional ES for the same total population size, further exploration of its scalability on distributed systems is implied.
  • Exploration of Other Transfer Mechanisms: The paper focuses on importance sampling; future work could explore other methods for inter-task skill transfer within the multitasking framework.

Summary updated

Abstract

Studies have shown evolution strategies (ES) to be a promising approach for reinforcement learning (RL) with deep neural networks. However, the issue of high sample complexity persists in applications of ES to deep RL over long horizons. This paper is the first to address the shortcoming of today's methods via a novel neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer information from a set of auxiliary tasks (of short episode length) to the target (full length) RL task at hand. The auxiliary tasks, extracted from the target, allow an agent to update and quickly evaluate policies on shorter time horizons. The evolved skills are then transferred to guide the longer and harder task towards an optimal policy. We demonstrate that the NuEMT algorithm achieves data-efficient evolutionary RL, reducing expensive agent-environment interaction data requirements. Our key algorithmic contribution in this setting is to introduce, for the first time, a multitask skills transfer mechanism based on the statistical importance sampling technique. In addition, an adaptive resource allocation strategy is utilized to assign computational resources to auxiliary tasks based on their gleaned usefulness. Experiments on a range of continuous control tasks from the OpenAI Gym confirm that our proposed algorithm is efficient compared to recent ES baselines.
BibTeX
@article{zhang2023lean,
  author = {Nick Zhang and Abhishek Gupta and Zefeng Chen and Yew-Soon Ong},
  title = {{Lean Evolutionary Reinforcement Learning by Multitasking with Importance Sampling}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=YRTbyKimtW},
}
Figure