TL;DR
This paper introduces NuEMT, a novel neuroevolutionary multitasking algorithm that significantly improves the data efficiency of evolution strategies (ES) for reinforcement learning (RL). By leveraging shorter "auxiliary" tasks to transfer learned skills to a main, longer task using importance sampling, NuEMT drastically reduces the amount of environment interaction data needed for training.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents learn by interacting with an environment and maximizing rewards. Evolution Strategies (ES) are a promising, derivative-free approach for training deep neural network policies in RL, especially for continuous control tasks. However, ES suffers from high "sample complexity," meaning it requires a vast amount of agent-environment interactions to learn effectively, particularly for tasks with long horizons (e.g., complex simulations or long-duration tasks). This makes training computationally expensive and time-consuming. Existing methods like "progressive episode lengths" (PEL) attempt to mitigate this by training on tasks with increasing episode lengths sequentially, but this can lead to negative transfer if skills learned on shorter tasks don't generalize well to longer ones.
Methods
The core innovation is the NuEMT (Neuroevolutionary Multitasking) algorithm. It builds upon the OpenAI-ES framework, which uses a population of policies and adds Gaussian noise to explore the policy space. NuEMT extends this by:
- Multitask Formulation: Instead of training a single task, NuEMT trains a target task alongside multiple "auxiliary" tasks concurrently. These auxiliary tasks are derived from the main task but have shorter episode lengths.
- Importance Sampling for Skill Transfer: The key novelty is using statistical importance sampling to transfer skills from auxiliary tasks to the main task. This allows the algorithm to learn from policies evolved on shorter episodes and use that knowledge to guide the search for the optimal policy on the longer, main task. This mechanism effectively "transfers" learned skills without the risk of negative transfer inherent in sequential approaches.
- Adaptive Resource Allocation: NuEMT dynamically allocates computational resources (population size) to auxiliary tasks based on their "usefulness" to the main task. Tasks that contribute more to the main task's learning receive more resources, while less useful tasks are de-emphasized, minimizing wasted computation.
- Probabilistic Mixture Model: The algorithm uses a mixture model to represent the combined probability distribution of candidate solutions from all tasks, allowing for controlled inter-task information exchange.
Datasets
The paper evaluates NuEMT on standard continuous control tasks from the OpenAI Gym suite, specifically:
- MuJoCo tasks: HalfCheetah-v2, Swimmer-v2, Hopper-v2, Ant-v2, Walker2d-v2, Humanoid-v2.
- Box2D simulators: BipedalWalker-v3 with varying leg lengths.
No new benchmarks were introduced; instead, the focus was on demonstrating efficiency on existing, well-established RL environments.
Key Insights and Ideas
- Multitasking as a Data Efficiency Booster: The central idea is that learning on simpler, shorter tasks can provide valuable "building blocks" of knowledge that accelerate learning on more complex, longer tasks. This is analogous to how humans learn by mastering simpler skills before tackling more advanced ones.
- Importance Sampling for Controlled Transfer: Using importance sampling is a clever way to bridge the gap between different task distributions (due to varying episode lengths) and enable effective skill transfer without explicit model rebuilding, as seen in some prior EMT approaches.
- Adaptive Resource Allocation is Crucial: Simply having auxiliary tasks isn't enough; dynamically assigning computational resources based on their actual contribution to the main task is key to maximizing efficiency and avoiding wasted effort on irrelevant skills.
- NuEMT Outperforms Sequential Transfer: The concurrent nature of multitasking in NuEMT appears to be more robust than sequential transfer methods like PEL, especially in scenarios where skills might not generalize perfectly across episode lengths.
Key Results
- Significant Data Efficiency Gains: NuEMT consistently outperforms the baseline OpenAI-ES and PEL methods in terms of sample efficiency across most tested MuJoCo and Box2D tasks. For example, in the HalfCheetah-v2 task, NuEMT achieved better performance with fewer timesteps compared to OpenAI-ES.
- Faster Convergence: NuEMT demonstrates accelerated convergence, reaching higher cumulative rewards earlier in the training process, particularly on the main task.
- Robustness: NuEMT shows strong performance across various tasks, including those with challenging dynamics like the Humanoid-v2 (where survival bonus was removed) and BipedalWalker-v3 with different leg lengths.
- Mixture Coefficient Learning Effectiveness: Experiments showed that learning and updating mixture coefficients (as implemented in NuEMT) leads to significantly better performance compared to using fixed mixture coefficients.
Limitations and Future Work
- Hyperparameter Sensitivity: While not explicitly detailed as a limitation, the paper mentions that the performance can be affected by the number of auxiliary tasks and the fixed population size. The optimal number of auxiliary tasks might be environment-dependent.
- Generalization Beyond RL: The authors suggest extending the general idea of leveraging simpler tasks to improve learning on complex problems beyond RL, such as in neural architecture search or other machine learning sub-fields.
- Hardware Efficiency: While NuEMT is computationally cheaper per iteration than conventional ES for the same total population size, further exploration of its scalability on distributed systems is implied.
- Exploration of Other Transfer Mechanisms: The paper focuses on importance sampling; future work could explore other methods for inter-task skill transfer within the multitasking framework.
Summary updated