Multitask Neuroevolution for Reinforcement Learning With Long and Short Episodes

Nick Zhang, Abhishek Gupta, Zefeng Chen, Yew-Soon Ong

TL;DR

This paper introduces NuEMT, a novel neuroevolutionary multitasking algorithm that significantly improves data efficiency in reinforcement learning by transferring skills learned from short, auxiliary tasks to a main, long-horizon task. By leveraging importance sampling and adaptive resource allocation, NuEMT achieves faster learning and better performance compared to existing methods.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn to make decisions in dynamic environments by maximizing cumulative rewards. While evolution strategies (ES) have emerged as a powerful, derivative-free approach for training deep neural network policies in RL, they often suffer from high sample complexity, requiring a vast amount of agent-environment interactions. This problem is particularly acute for tasks with long horizons (i.e., long episodes), where evaluating a single policy can be computationally expensive. Existing methods like Progressive Episode Lengths (PEL) attempt to address this by sequentially training on tasks of increasing episode lengths, but this can lead to negative transfer if skills learned on shorter tasks don't generalize well to longer ones. The paper aims to solve this by enabling efficient skill transfer from multiple short auxiliary tasks to a single long target task concurrently, while mitigating the risk of negative transfer.

Methods

The core innovation is the Neuroevolutionary Multitasking (NuEMT) algorithm. NuEMT builds upon the OpenAI-ES framework, which uses a population of policies and adds Gaussian noise to explore the policy space. The key novelties are:

  • Multitask Formulation: Instead of sequential training, NuEMT trains the target task and multiple auxiliary tasks simultaneously. Auxiliary tasks are derived from the main task but have shorter episode lengths.
  • Importance Sampling for Skill Transfer: NuEMT uses a statistical importance sampling technique to transfer knowledge from auxiliary tasks to the main task. This allows the algorithm to learn from the successes of policies trained on shorter episodes and apply that knowledge to the longer, more challenging task. This mechanism is designed to be more robust than simple solution seeding.
  • Adaptive Resource Allocation: The algorithm dynamically allocates computational resources (population size) to auxiliary tasks based on their usefulness to the target task. Tasks that provide valuable transferable skills receive more resources, while less useful tasks receive fewer, minimizing wasted computation.
  • Mixture Model: A mixture model is used to represent the probabilistic relationship between tasks, allowing for controlled and adaptive skill transfer.

Datasets

The paper evaluates NuEMT on continuous control tasks from the OpenAI Gym benchmark suite. Specifically, it uses:

  • MuJoCo tasks: HalfCheetah-v2, Swimmer-v2, Hopper-v2, Ant-v2, Walker2d-v2, and Humanoid-v2.
  • Box2D simulators: BipedalWalker-v3 with varying leg lengths (0.5x, 1.0x, and 1.5x the default).

No new benchmarks were introduced; the focus was on demonstrating NuEMT's effectiveness on existing, challenging RL environments.

Key Insights and Ideas

  • Concurrent Multitasking is Superior to Sequential Transfer: The paper strongly suggests that training multiple auxiliary tasks simultaneously with the target task (multitasking) is more effective than sequentially training on tasks of increasing difficulty (like PEL). This is because concurrent training allows for more dynamic adaptation and avoids the pitfalls of negative transfer that can occur when skills are rigidly propagated.
  • Importance Sampling as a "Smart Filter" for Skill Transfer: The use of importance sampling is a clever way to leverage the successes of auxiliary tasks without blindly copying their policies. It allows the algorithm to weigh the contributions of different tasks based on how well their learned skills actually benefit the target task.
  • Adaptive Resource Allocation Prevents Wasted Effort: Dynamically assigning population sizes to auxiliary tasks is crucial for efficiency. It ensures that computational resources are focused on tasks that are genuinely contributing to the learning process, rather than being spread thinly across all tasks regardless of their utility.
  • Shorter Episodes are Valuable "Training Grounds": The core idea that learning from shorter, simpler tasks can accelerate learning on longer, more complex tasks is validated. This is intuitive, as shorter episodes allow for faster policy updates and exploration of basic skills.

Key Results

  • Data Efficiency: NuEMT significantly reduces the amount of agent-environment interaction data required to achieve a target performance level compared to OpenAI-ES and PEL. For example, in the HalfCheetah-v2 task, NuEMT achieved a higher reward with fewer timesteps than both baselines.
  • Performance Improvement: NuEMT consistently outperformed the baseline algorithms on most of the tested MuJoCo tasks, demonstrating its ability to achieve better final performance.
  • Faster Convergence: The NuEMT algorithm showed faster convergence trends, reaching higher rewards earlier in the training process, particularly in the initial stages of evolution.
  • Robustness: NuEMT demonstrated better performance on more challenging Box2D tasks (BipedalWalker-v3 with longer leg lengths) where PEL struggled.
  • Impact of Auxiliary Tasks: Experiments showed that the number of auxiliary tasks matters. With a fixed population size, NuEMT with 3 auxiliary tasks performed best on BipedalWalker-v3, suggesting an optimal balance between exploration and exploitation.

Limitations and Future Work

  • Limitations:
    • The paper notes that when the gap between the parameter distributions of different tasks becomes very large, the importance weights can approach zero, potentially hindering transfer. They address this with a projection technique, but it's an acknowledged challenge.
  • While NuEMT is more data-efficient, the computational cost per iteration might still be high for extremely complex environments or a very large number of tasks, although the paper argues it's more efficient than naive multitasking.
  • Future Work:
    • The authors plan to extend the general idea of leveraging simpler tasks to improve learning on complex problems beyond RL, into areas like neural architecture search.
  • They suggest that other machine learning sub-fields could benefit from jointly evolving multiple tasks to produce diverse models specialized for different datasets or hardware constraints.

Summary updated

Abstract

Studies have shown evolution strategies (ES) to be a promising approach for reinforcement learning (RL) with deep neural networks. However, the issue of high sample complexity persists in applications of ES to deep RL over long horizons. This article is the first to address the shortcoming of today’s methods via a novel neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer information from a set of auxiliary tasks (of short episode length) to the target (full length) RL task at hand. The auxiliary tasks, extracted from the target, allow an agent to update and quickly evaluate policies on shorter time horizons. The evolved skills are then transferred to guide the longer and harder task toward an optimal policy. We demonstrate that the NuEMT algorithm achieves data-efficient evolutionary RL, reducing expensive agent-environment interaction data requirements. Our key algorithmic contribution in this setting is to introduce a first multitask skills transfer mechanism based on the statistical importance sampling technique. In addition, an adaptive resource allocation strategy is utilized to assign computational resources to auxiliary tasks based on their gleaned usefulness. Experiments on a range of continuous control tasks from the OpenAI Gym confirm that our proposed algorithm is efficient compared to recent ES baselines.
BibTeX
@article{zhang2023multitask,
  author = {Nick Zhang and Abhishek Gupta and Zefeng Chen and Yew-Soon Ong},
  title = {{Multitask Neuroevolution for Reinforcement Learning With Long and Short Episodes}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=YRTbyKimtW},
}
Figure