TL;DR
This paper introduces "Genetic Curriculum" (GC), a novel reinforcement learning (RL) training method that uses genetic algorithms to automatically generate challenging yet learnable training scenarios. GC significantly improves agent robustness, making them 2-8x less likely to fail in critical situations without sacrificing overall performance, by creating curricula that adapt to the agent's learning progress.
What Problem Does This Paper Address?
Training deep reinforcement learning (RL) agents for safety-critical applications, like autonomous vehicles, is challenging because achieving robust performance is paramount. Agents often struggle with a "long tail" of difficult scenarios that are rarely encountered during standard training, leading them to converge to suboptimal policies. Existing methods like adversarial training can create difficult scenarios but often require expert supervision to fine-tune and can quickly generate scenarios too hard for the agent to learn from. Other curriculum learning approaches are limited by fixed-length scenario encodings, hindering their ability to adapt to complex environments and evolving agent capabilities. This paper aims to bridge this gap by developing an automated, adaptive curriculum generation method that fosters robust RL agents.
Methods
The core innovation is the Genetic Curriculum (GC) algorithm. GC employs a genetic algorithm to evolve a population of training scenarios. Here's how it works:
- Scenario Encoding: Scenarios are represented as non-fixed length sequences, allowing for complex and dynamic descriptions. This is a key departure from prior work that relied on fixed-length encodings.
- Population Initialization: A population of random scenarios is initialized.
- Evaluation: The current RL agent's policy is used to evaluate each scenario in the population. Scenarios where the agent fails are prioritized.
- Crossover and Mutation: A genetic algorithm is used to generate new scenarios.
- Crossover: Scenarios are combined by mixing parts of their encodings. The probability of a scenario being selected as a parent is inversely proportional to its length, encouraging the retention of critical failure-inducing elements.
- Mutation: Random perturbations are introduced to existing scenarios to create new variations.
- Curriculum Generation: The genetic algorithm iteratively refines the population of scenarios, aiming to create a curriculum of increasingly challenging but learnable situations. The algorithm dynamically adjusts the length and complexity of scenarios as the agent's performance improves.
- Training Loop: The genetic algorithm runs in parallel to the RL agent's training. At the end of each training epoch, the agent's policy is fixed, and the genetic algorithm generates a new set of scenarios for the next epoch. This separation makes GC computationally more efficient than methods that nest training within the genetic algorithm.
Datasets
The experiments were conducted on several standard RL benchmarks, modified to include failure conditions:
- Bipedal Walker (Hardcore/System): Tests robustness against various obstacles and simulated system failures.
- Lunar Lander: Assesses the agent's ability to land safely, even with limited engine power due to failures.
- Hopper and Walker: These benchmarks involve legged robots with simulated payloads, where failures can occur in servo torques.
The paper also mentions using "randomly generated scenarios" for testing, implying that the evaluation sets are diverse and representative of potential failure modes.
Key Insights and Ideas
- Adaptive Curriculum is Crucial for Robustness: The paper strongly suggests that simply generating difficult scenarios isn't enough; the curriculum needs to be adaptive and provide supporting scenarios to facilitate learning.
- Non-Fixed Length Encoding is a Game-Changer: The ability to use non-fixed length scenario encodings allows GC to adapt to the agent's evolving performance and generate much more complex scenarios than previous methods. This reduces the need for expert supervision in designing scenario representations.
- Genetic Algorithms as Scenario Generators: Using genetic algorithms for scenario generation is effective because crossover and mutation can create a diverse yet related set of scenarios, promoting skill transfer and efficient exploration of the scenario space.
- Separating Curriculum Generation from Training: Running the genetic algorithm for curriculum generation separately from the agent's training loop significantly improves computational efficiency compared to methods like POET.
- Robustness is Hard to Capture with Reward Alone: The results show that while rewards might improve, failure rates can persist. This highlights the importance of directly optimizing for robustness and using metrics beyond just cumulative reward.
Key Results
- Significant Robustness Improvement: Agents trained with GC were 2-8x less likely to fail compared to those trained with baseline RL (SAC/TD3) and other state-of-the-art methods.
- Competitive Performance: GC achieved competitive or superior rewards compared to baseline methods, demonstrating that robustness gains did not come at the cost of overall performance.
- Efficient Learning: Despite an initial overhead for running the genetic algorithm, GC's ability to provide better training examples led to sustained performance improvement and a better final solution compared to methods where marginal utility per environment interaction diminishes quickly.
- Ablation Study Insights:
- Crossover is Key: Removing crossover significantly degraded performance, indicating the importance of mixing scenario elements.
- Mutation Aids Exploration: While less critical than crossover, mutation helped in exploring a wider scenario space.
- Similarity Matters: Curricula with similar scenarios (low genetic distance) generally performed better, but a balance was needed to ensure sufficient coverage of the scenario space.
Limitations and Future Work
- Computational Load: While more efficient than some prior methods, GC still incurs computational overhead due to the genetic algorithm. Future work aims to decrease this load.
- Convergence Rate: The paper notes that GC is a bit slow at the start. Improving the rate of convergence is a stated future goal.
- Real-World Complexity: The current experiments are on simulated benchmarks. Applying GC to real-world, more complex scenarios is a promising direction.
- General Skillset: The paper suggests that robust training should focus on learning a general skillset applicable across tasks, which could be an area for future exploration.
Summary updated